2024-03-25T16:10:10.976 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:10:10.976 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/orig.config.yaml 2024-03-25T16:10:11.019 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:10:11.118 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621252' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK6Y5chr1P3W+bPK6Xaeaip+/nsTZQ5qzyhjymkTeb+IGibowaAwcZ9ftmZrfeqMR0LnOXCZWuKsj9ExgiCMKQ= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfSSRsp6KvzxWvC/Wea72U/E4M7bNUtEnuPpOpE7RjfVCjbi2dR1vm7niezXKK3+lqWOkBM70RbsK8dCV/pf4I= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:10:11.118 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:10:11.119 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:10:11.119 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:10:11.120 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:10:11.145 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:10:11.145 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:10:11.145 INFO:teuthology.packaging:ref: None 2024-03-25T16:10:11.146 INFO:teuthology.packaging:tag: None 2024-03-25T16:10:11.146 INFO:teuthology.packaging:branch: main 2024-03-25T16:10:11.146 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:11.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:10:11.463 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:10:11.465 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:10:11.472 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:10:11.472 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:10:11.505 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:10:11.522 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:10:11.580 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:10:11.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:04:44.478006', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK6Y5chr1P3W+bPK6Xaeaip+/nsTZQ5qzyhjymkTeb+IGibowaAwcZ9ftmZrfeqMR0LnOXCZWuKsj9ExgiCMKQ='} 2024-03-25T16:10:11.624 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:04:44.479216', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfSSRsp6KvzxWvC/Wea72U/E4M7bNUtEnuPpOpE7RjfVCjbi2dR1vm7niezXKK3+lqWOkBM70RbsK8dCV/pf4I='} 2024-03-25T16:10:11.624 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:10:11.651 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:10:11.652 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:10:11.652 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:10:11.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f71386f60d0>, signals=[15]) 2024-03-25T16:10:11.780 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:10:11.787 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:10:11.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T16:10:11.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:11.877 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T16:10:11.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:11.967 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:10:11.974 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-25T16:10:11.992 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-25T16:10:11.992 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-25T16:10:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-25T16:10:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-25T16:10:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-25T16:10:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:12.049 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:12.050 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-25T16:10:12.072 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-03-25T16:10:12.088 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-03-25T16:10:12.089 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:12.146 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:12.147 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:12.147 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:12.147 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-03-25T16:10:12.171 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:10:12.181 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:10:12.222 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:10:12.222 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:12.226 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:12.243 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:10:12.295 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:10:12.296 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:12.298 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:12.331 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:10:12.397 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:10:12.421 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:10:12.421 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:10:12.422 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:12.422 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:12.438 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-25T16:10:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:12.440 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-25T16:10:12.495 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:12.496 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-25T16:10:12.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-25T16:10:12.497 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:12.497 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-03-25T16:10:12.497 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-03-25T16:10:14.579 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:10:14.663 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:10:16.204 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:10:16.297 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T16:10:17.086 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:10:17.464 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-25T16:10:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T16:10:22.647 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:10:24.080 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-25T16:10:28.048 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 21 MB 00:03 2024-03-25T16:10:33.131 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2024-03-25T16:10:34.170 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-25T16:10:35.221 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:10:35.221 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:10:35.320 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:10:35.327 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:10:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-25T16:10:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:10:35.329 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 76 M 2024-03-25T16:10:35.329 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-25T16:10:35.329 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:10:36.049 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 66 kB/s | 44 kB 00:00 2024-03-25T16:10:46.611 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 704 kB/s | 6.1 MB 00:08 2024-03-25T16:10:48.388 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-25T16:10:50.507 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T16:10:55.488 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:10:57.636 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:10:57.636 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:10:57.723 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:10:57.739 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-03-25T16:10:57.740 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:10:57.741 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-03-25T16:10:57.741 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-03-25T16:10:57.741 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:10:58.456 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 108 kB/s | 44 kB 00:00 2024-03-25T16:11:00.623 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 5.8 MB/s | 15 MB 00:02 2024-03-25T16:11:01.238 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 590 kB/s | 15 MB 00:25 2024-03-25T16:11:11.846 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 2.4 MB/s | 33 MB 00:13 2024-03-25T16:11:16.578 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 694 kB/s | 27 MB 00:40 2024-03-25T16:11:23.679 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.1 MB/s | 27 MB 00:25 2024-03-25T16:11:23.679 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:23.679 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.9 MB/s | 76 MB 00:25 2024-03-25T16:11:24.347 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:11:24.757 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:11:24.757 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:11:25.771 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:11:25.771 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:11:28.232 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:11:28.907 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:28.943 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 639 kB/s | 33 MB 00:53 2024-03-25T16:11:28.943 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:28.943 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.4 MB/s | 76 MB 00:53 2024-03-25T16:11:28.995 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:29.603 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:11:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:11:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:11:31.004 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:11:31.004 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:11:31.745 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:31.881 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:33.469 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:11:34.115 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:34.208 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:36.601 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:36.647 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:37.192 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:41.892 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:41.991 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:46.733 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:10.720 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:10.720 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:12:10.720 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:12:10.765 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:11.138 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:11.138 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:11.138 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:11.139 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:12:11.589 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:12:11.921 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:12.024 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:12.025 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:12.026 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:12:12.026 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:12:12.026 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:12:12.026 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:12:12.026 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:12:12.026 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:12.106 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:12.106 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:12.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-03-25T16:12:12.141 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:12:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:12:12.143 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:12:12.143 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:12:12.144 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:12.799 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-03-25T16:12:14.174 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:14.203 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-03-25T16:12:14.205 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-25T16:12:14.221 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.aDiN1iB6bO 2024-03-25T16:12:14.221 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aDiN1iB6bO 2024-03-25T16:12:14.296 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.aDiN1iB6bO 2024-03-25T16:12:14.441 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.aDiN1iB6bO is 6KB 2024-03-25T16:12:14.488 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.aDiN1iB6bO 2024-03-25T16:12:14.503 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:14.575 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:14.575 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:14.575 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:14.575 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:14.577 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:14.727 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-03-25T16:12:15.979 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:15.979 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:12:15.979 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:12:16.022 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.377 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.377 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:16.377 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:16.377 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:16.767 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.767 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:12:16.768 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:12:17.092 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:17.505 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:17.506 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.506 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.506 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:12:17.506 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:12:17.506 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:12:17.506 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:12:17.506 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:17.587 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:17.587 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-25T16:12:17.621 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-25T16:12:17.621 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-25T16:12:17.622 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:12:17.623 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:12:17.624 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:12:17.624 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:18.213 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-25T16:12:19.500 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:19.537 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-25T16:12:19.539 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-25T16:12:19.554 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.e3lWwXLvEG 2024-03-25T16:12:19.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.e3lWwXLvEG 2024-03-25T16:12:19.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.e3lWwXLvEG 2024-03-25T16:12:19.768 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.e3lWwXLvEG is 6KB 2024-03-25T16:12:19.815 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.e3lWwXLvEG 2024-03-25T16:12:19.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:19.894 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:19.894 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:19.895 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:19.895 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:19.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:20.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-25T16:12:44.749 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:12:44.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:12:44.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:50.054 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:12:50.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:12:50.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:03.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-25T16:13:08.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-25T16:13:12.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:13:12.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:15.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-25T16:13:17.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:13:17.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:20.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-25T16:13:27.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:13:27.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:32.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:13:32.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:51.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-25T16:14:06.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:14:06.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:06.622 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-25T16:14:07.122 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:14:07.122 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:07.123 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-25T16:14:07.137 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:07.138 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:07.138 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:07.138 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:08.141 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:14:08.141 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:08.225 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-25T16:14:08.260 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:27.214 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:14:42.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:14:42.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:42.631 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-25T16:14:43.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:14:43.003 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:43.003 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-25T16:14:43.018 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:43.018 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:43.018 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:43.019 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:44.020 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:14:44.020 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:44.091 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-03-25T16:14:44.135 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:44.135 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:14:44.143 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:14:44.143 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:44.147 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:44.164 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:14:44.171 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:14:44.244 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:14:44.244 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:44.246 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:44.303 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:14:44.311 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:14:44.311 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:44.320 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:44.361 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:44.368 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:44.374 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:44.382 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:44.384 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:14:44.391 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:14:44.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:44.393 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:44.446 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:14:44.456 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:14:44.456 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:44.464 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:44.503 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:44.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:44.618 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:44.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:44.743 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T16:14:44.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:44.811 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:44.856 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:44.912 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:44.997 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:45.041 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T16:14:45.041 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:45.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-25T16:14:45.112 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-03-25T16:14:45.155 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:45.203 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:45.623 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:14:45.632 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:14:45.632 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:14:45.641 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:14:45.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-25T16:14:45.712 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-25T16:14:45.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-03-25T16:14:45.762 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-03-25T16:14:45.764 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:14:45.764 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-25T16:14:45.785 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-25T16:14:45.785 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-03-25T16:14:45.805 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-03-25T16:14:45.806 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi176.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:14:45.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:45.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:45.883 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:14:45.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:45.930 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:45.974 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:14:45.985 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:14:45.990 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:14:46.118 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:14:46.134 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:14:46.135 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi176.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:18:58.564 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi176.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-03-25T16:18:58.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:18:58.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:58.655 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-25T16:18:58.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-25T16:18:58.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:18:58.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:58.826 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-25T16:18:58.902 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-25T16:18:58.903 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:18:58.914 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:18:58.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:58.914 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:58.917 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:58.917 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:58.945 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:58.962 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:58.976 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:58.991 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-25T16:18:58.993 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:59.004 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-25T16:18:59.021 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:59.023 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-03-25T16:18:59.036 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-03-25T16:18:59.037 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:59.053 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:59.069 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:59.092 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:59.096 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:59.096 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-25T16:18:59.096 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.096 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.096 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.097 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.118 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:59.121 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:59.122 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-25T16:18:59.122 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.122 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.122 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.122 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:59.123 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:18:59.132 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:18:59.132 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:18:59.132 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:18:59.132 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:18:59.138 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:18:59.138 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:18:59.138 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:59.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:59.141 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:59.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:59.283 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:59.284 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:59.288 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:59.288 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:59.615 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:59.615 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T16:18:59.616 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:59.645 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:59.645 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:59.676 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:59.677 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T16:18:59.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:59.706 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:59.706 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:59.713 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:59.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:59.792 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:59.855 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:59.859 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-03-25T16:18:59.860 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-03-25T16:18:59.922 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-03-25T16:18:59.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-03-25T16:19:00.103 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-03-25T16:19:00.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:00.155 INFO:teuthology.orchestra.run.smithi176.stdout:74 files removed 2024-03-25T16:19:00.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:01.421 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 87 kB/s | 82 kB 00:00 2024-03-25T16:19:01.507 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T16:19:02.296 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:19:02.307 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:19:03.158 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:19:03.188 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:19:03.657 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:19:03.919 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-25T16:19:05.216 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-25T16:19:05.619 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T16:19:15.517 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 860 kB/s | 8.0 MB 00:09 2024-03-25T16:19:16.622 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 742 kB/s | 8.0 MB 00:11 2024-03-25T16:19:17.443 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-25T16:19:21.961 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.1 MB 00:01 2024-03-25T16:19:24.395 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T16:19:25.774 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:19:29.823 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-25T16:19:30.763 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T16:19:33.283 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:19:33.298 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:19:33.299 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:19:33.300 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:Install 30 Packages 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 58 M 2024-03-25T16:19:33.301 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:19:35.082 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-25T16:19:36.633 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:19:37.929 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T16:19:40.714 INFO:teuthology.orchestra.run.smithi186.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 747 kB/s | 5.3 MB 00:07 2024-03-25T16:19:40.822 INFO:teuthology.orchestra.run.smithi186.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-25T16:19:41.797 INFO:teuthology.orchestra.run.smithi186.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 761 kB/s | 742 kB 00:00 2024-03-25T16:19:42.639 INFO:teuthology.orchestra.run.smithi186.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 592 kB/s | 498 kB 00:00 2024-03-25T16:19:42.818 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-25T16:19:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 842 kB/s | 9.7 MB 00:11 2024-03-25T16:19:45.265 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:19:45.279 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:19:45.279 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:19:45.279 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:19:45.279 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:19:45.279 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:19:45.280 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout:Install 30 Packages 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-03-25T16:19:45.282 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:19:45.283 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-03-25T16:19:45.283 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:19:45.440 INFO:teuthology.orchestra.run.smithi186.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T16:19:45.631 INFO:teuthology.orchestra.run.smithi186.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 686 kB/s | 131 kB 00:00 2024-03-25T16:19:45.848 INFO:teuthology.orchestra.run.smithi186.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 747 kB/s | 162 kB 00:00 2024-03-25T16:19:46.423 INFO:teuthology.orchestra.run.smithi186.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 557 kB/s | 320 kB 00:00 2024-03-25T16:19:46.890 INFO:teuthology.orchestra.run.smithi186.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 638 kB/s | 298 kB 00:00 2024-03-25T16:19:47.091 INFO:teuthology.orchestra.run.smithi186.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 497 kB/s | 99 kB 00:00 2024-03-25T16:19:47.249 INFO:teuthology.orchestra.run.smithi186.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 669 kB/s | 106 kB 00:00 2024-03-25T16:19:47.349 INFO:teuthology.orchestra.run.smithi186.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-25T16:19:47.391 INFO:teuthology.orchestra.run.smithi186.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:19:47.467 INFO:teuthology.orchestra.run.smithi186.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T16:19:47.508 INFO:teuthology.orchestra.run.smithi186.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T16:19:47.542 INFO:teuthology.orchestra.run.smithi186.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:19:47.759 INFO:teuthology.orchestra.run.smithi186.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:19:47.818 INFO:teuthology.orchestra.run.smithi186.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-25T16:19:47.876 INFO:teuthology.orchestra.run.smithi186.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-25T16:19:48.027 INFO:teuthology.orchestra.run.smithi186.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-03-25T16:19:48.277 INFO:teuthology.orchestra.run.smithi186.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 926 kB/s | 5.1 MB 00:05 2024-03-25T16:19:48.352 INFO:teuthology.orchestra.run.smithi186.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-03-25T16:19:48.428 INFO:teuthology.orchestra.run.smithi186.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-25T16:19:48.494 INFO:teuthology.orchestra.run.smithi186.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T16:19:48.661 INFO:teuthology.orchestra.run.smithi186.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-25T16:19:48.720 INFO:teuthology.orchestra.run.smithi186.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T16:19:48.871 INFO:teuthology.orchestra.run.smithi186.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T16:19:49.096 INFO:teuthology.orchestra.run.smithi186.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-03-25T16:19:51.554 INFO:teuthology.orchestra.run.smithi186.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-03-25T16:19:54.154 INFO:teuthology.orchestra.run.smithi186.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 633 kB/s | 3.1 MB 00:05 2024-03-25T16:19:54.535 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 606 kB/s | 5.3 MB 00:09 2024-03-25T16:19:54.651 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T16:19:56.943 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 324 kB/s | 742 kB 00:02 2024-03-25T16:19:58.410 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 340 kB/s | 498 kB 00:01 2024-03-25T16:20:00.468 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 1.3 MB/s | 20 MB 00:14 2024-03-25T16:20:01.043 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 78 kB/s | 45 kB 00:00 2024-03-25T16:20:01.201 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 635 kB/s | 9.7 MB 00:15 2024-03-25T16:20:01.419 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 350 kB/s | 131 kB 00:00 2024-03-25T16:20:01.544 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 472 kB/s | 162 kB 00:00 2024-03-25T16:20:02.194 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 413 kB/s | 320 kB 00:00 2024-03-25T16:20:02.445 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-25T16:20:02.512 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 308 kB/s | 298 kB 00:00 2024-03-25T16:20:02.570 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 845 kB/s | 106 kB 00:00 2024-03-25T16:20:02.620 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2024-03-25T16:20:02.646 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-25T16:20:02.679 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:20:02.705 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-25T16:20:02.730 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-25T16:20:02.813 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 419 kB/s | 35 kB 00:00 2024-03-25T16:20:02.880 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 670 kB/s | 44 kB 00:00 2024-03-25T16:20:02.906 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-25T16:20:03.206 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 950 kB/s | 309 kB 00:00 2024-03-25T16:20:03.281 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-25T16:20:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-25T16:20:03.398 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T16:20:03.540 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-25T16:20:03.607 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:20:03.674 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-03-25T16:20:03.816 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-25T16:20:06.671 INFO:teuthology.orchestra.run.smithi186.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 612 kB/s | 20 MB 00:33 2024-03-25T16:20:06.675 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:06.675 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.7 MB/s | 58 MB 00:33 2024-03-25T16:20:06.707 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:20:06.745 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:20:06.746 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:20:07.213 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:20:07.213 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:20:08.672 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:20:08.764 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:20:08.791 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:20:08.817 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:20:08.848 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:08.874 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:09.048 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:09.104 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:09.416 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 602 kB/s | 3.4 MB 00:05 2024-03-25T16:20:09.432 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:20:09.486 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:09.568 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:09.656 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:20:09.679 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:09.741 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:09.749 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 540 kB/s | 3.1 MB 00:05 2024-03-25T16:20:09.776 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:20:09.815 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:20:09.882 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:09.906 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:09.969 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:09.996 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:20:10.147 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:20:10.188 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:20:10.220 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:20:10.249 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:20:10.282 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:20:10.591 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:20:10.620 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:10.693 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:10.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:20:10.768 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:20:11.183 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 409 kB/s | 5.1 MB 00:12 2024-03-25T16:20:11.188 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:11.188 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.2 MB/s | 58 MB 00:25 2024-03-25T16:20:11.220 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:20:11.222 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:20:11.248 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:20:11.319 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:11.386 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:20:11.417 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:20:11.740 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:20:11.740 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:20:13.191 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:20:13.274 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:20:13.313 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:20:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:20:13.369 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:13.401 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:13.506 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:13.547 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:13.624 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:13.673 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:13.992 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:20:14.048 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:14.105 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:14.141 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:14.146 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:14.220 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:14.220 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:20:14.220 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:20:14.220 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:14.234 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:20:14.251 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:14.257 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:14.329 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:14.357 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:20:14.401 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:20:14.468 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:14.493 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:14.556 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:20:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:20:14.726 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:20:14.766 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:20:14.799 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:20:14.828 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:20:14.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:20:15.172 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:20:15.199 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:20:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:20:15.812 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:20:15.843 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:15.914 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:15.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:20:16.012 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:20:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:18.242 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:18.830 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:18.870 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:18.949 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:18.949 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:20:18.949 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:20:18.949 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:18.979 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:23.216 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:23.216 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-03-25T16:20:23.216 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-03-25T16:20:23.217 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:24.142 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:24.167 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:24.168 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:24.263 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:24.263 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:20:25.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:20:25.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:20:25.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:20:25.364 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:20:26.464 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:20:26.465 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:20:26.466 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:20:26.467 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:20:26.467 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:20:26.467 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:20:26.467 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:26.467 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:20:26.707 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-03-25T16:20:27.271 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:20:27.427 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:20:27.427 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:20:27.427 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:20:27.427 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:20:27.427 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-03-25T16:20:27.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:20:27.429 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 48 M 2024-03-25T16:20:27.429 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 204 M 2024-03-25T16:20:27.429 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:20:27.858 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:20:27.934 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-25T16:20:27.967 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-03-25T16:20:28.088 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-03-25T16:20:28.089 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-03-25T16:20:28.089 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:28.184 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 864 kB/s | 187 kB 00:00 2024-03-25T16:20:28.693 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 164 kB/s | 163 kB 00:00 2024-03-25T16:20:29.014 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:29.038 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:20:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:29.039 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:29.041 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:29.133 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:29.133 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:30.261 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:30.261 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:20:30.261 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:20:30.261 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:20:30.261 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:20:30.262 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:20:30.263 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:20:30.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:20:31.291 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.292 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:20:31.293 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:20:31.294 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:31.294 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:20:31.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-03-25T16:20:32.216 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:50 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:20:32.370 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:20:32.371 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:20:32.371 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:20:32.371 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:20:32.371 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:32.372 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:20:32.373 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:20:32.373 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-25T16:20:32.373 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:20:32.373 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 48 M 2024-03-25T16:20:32.374 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 204 M 2024-03-25T16:20:32.374 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:20:32.763 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-25T16:20:32.863 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T16:20:32.888 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T16:20:33.080 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 980 kB/s | 187 kB 00:00 2024-03-25T16:20:33.472 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 197 kB/s | 163 kB 00:00 2024-03-25T16:21:33.245 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 739 kB/s | 47 MB 01:05 2024-03-25T16:21:33.246 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:33.246 INFO:teuthology.orchestra.run.smithi186.stdout:Total 750 kB/s | 48 MB 01:05 2024-03-25T16:21:33.258 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:21:33.278 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:21:33.278 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:21:33.653 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:21:33.653 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:21:34.260 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:21:34.327 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:21:34.378 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:21:34.418 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:34.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:21:34.489 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:34.772 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 779 kB/s | 47 MB 01:02 2024-03-25T16:21:34.775 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:34.775 INFO:teuthology.orchestra.run.smithi176.stdout:Total 791 kB/s | 48 MB 01:02 2024-03-25T16:21:34.788 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:21:34.808 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:21:34.808 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:21:35.184 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:21:35.185 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:21:35.777 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:21:35.864 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:21:35.900 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:21:35.932 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:35.977 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:21:36.000 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:38.951 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:39.022 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:40.470 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:40.541 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:41.369 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:41.369 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:21:41.369 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:21:41.369 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:41.369 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:21:41.370 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:21:41.778 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:21:41.779 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:21:42.113 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-03-25T16:21:42.682 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:21:42.840 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:21:42.844 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:21:42.847 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-25T16:21:42.847 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:21:42.847 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-25T16:21:42.847 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:21:42.848 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:21:42.849 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-25T16:21:42.850 INFO:teuthology.orchestra.run.smithi186.stdout:Install 36 Packages 2024-03-25T16:21:42.851 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:21:42.851 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 30 M 2024-03-25T16:21:42.851 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 107 M 2024-03-25T16:21:42.851 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:21:43.220 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:21:43.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-03-25T16:21:43.534 INFO:teuthology.orchestra.run.smithi186.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:21:44.097 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:21:44.254 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:21:44.256 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:21:44.257 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:21:44.258 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout:Install 36 Packages 2024-03-25T16:21:44.259 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:21:44.260 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 30 M 2024-03-25T16:21:44.260 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 M 2024-03-25T16:21:44.260 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:21:44.834 INFO:teuthology.orchestra.run.smithi186.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 911 kB/s | 1.5 MB 00:01 2024-03-25T16:21:44.991 INFO:teuthology.orchestra.run.smithi176.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-25T16:21:45.851 INFO:teuthology.orchestra.run.smithi186.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 808 kB/s | 2.1 MB 00:02 2024-03-25T16:21:46.025 INFO:teuthology.orchestra.run.smithi176.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-03-25T16:21:46.143 INFO:teuthology.orchestra.run.smithi186.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 849 kB/s | 247 kB 00:00 2024-03-25T16:21:46.260 INFO:teuthology.orchestra.run.smithi186.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:21:46.302 INFO:teuthology.orchestra.run.smithi186.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:21:46.327 INFO:teuthology.orchestra.run.smithi186.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:21:46.353 INFO:teuthology.orchestra.run.smithi186.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:21:46.378 INFO:teuthology.orchestra.run.smithi186.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:21:46.420 INFO:teuthology.orchestra.run.smithi186.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:21:46.487 INFO:teuthology.orchestra.run.smithi186.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:21:46.521 INFO:teuthology.orchestra.run.smithi186.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:21:46.563 INFO:teuthology.orchestra.run.smithi186.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:21:46.613 INFO:teuthology.orchestra.run.smithi186.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-25T16:21:46.747 INFO:teuthology.orchestra.run.smithi186.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-25T16:21:46.797 INFO:teuthology.orchestra.run.smithi186.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-03-25T16:21:47.117 INFO:teuthology.orchestra.run.smithi176.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 4.7 MB 00:02 2024-03-25T16:21:47.130 INFO:teuthology.orchestra.run.smithi186.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 453 kB/s | 151 kB 00:00 2024-03-25T16:21:47.206 INFO:teuthology.orchestra.run.smithi186.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 581 kB/s | 43 kB 00:00 2024-03-25T16:21:47.284 INFO:teuthology.orchestra.run.smithi176.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-25T16:21:47.306 INFO:teuthology.orchestra.run.smithi186.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:21:47.401 INFO:teuthology.orchestra.run.smithi176.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:21:47.431 INFO:teuthology.orchestra.run.smithi186.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T16:21:47.443 INFO:teuthology.orchestra.run.smithi176.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:21:47.485 INFO:teuthology.orchestra.run.smithi176.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-25T16:21:47.498 INFO:teuthology.orchestra.run.smithi186.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-25T16:21:47.518 INFO:teuthology.orchestra.run.smithi176.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:21:47.552 INFO:teuthology.orchestra.run.smithi176.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:21:47.565 INFO:teuthology.orchestra.run.smithi186.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T16:21:47.594 INFO:teuthology.orchestra.run.smithi176.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:21:47.632 INFO:teuthology.orchestra.run.smithi186.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T16:21:47.661 INFO:teuthology.orchestra.run.smithi176.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:21:47.699 INFO:teuthology.orchestra.run.smithi186.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T16:21:47.703 INFO:teuthology.orchestra.run.smithi176.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:21:47.753 INFO:teuthology.orchestra.run.smithi176.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:21:47.766 INFO:teuthology.orchestra.run.smithi186.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:21:47.803 INFO:teuthology.orchestra.run.smithi176.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T16:21:47.833 INFO:teuthology.orchestra.run.smithi186.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 697 kB/s | 46 kB 00:00 2024-03-25T16:21:47.900 INFO:teuthology.orchestra.run.smithi186.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T16:21:47.983 INFO:teuthology.orchestra.run.smithi186.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T16:21:48.042 INFO:teuthology.orchestra.run.smithi186.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T16:21:48.117 INFO:teuthology.orchestra.run.smithi186.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:21:48.176 INFO:teuthology.orchestra.run.smithi186.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-25T16:21:48.260 INFO:teuthology.orchestra.run.smithi186.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:21:48.352 INFO:teuthology.orchestra.run.smithi186.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.6 MB/s | 427 kB 00:00 2024-03-25T16:21:48.435 INFO:teuthology.orchestra.run.smithi186.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T16:21:48.612 INFO:teuthology.orchestra.run.smithi176.stdout:(15/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 540 kB/s | 2.1 MB 00:04 2024-03-25T16:21:48.637 INFO:teuthology.orchestra.run.smithi176.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 150 kB/s | 125 kB 00:00 2024-03-25T16:21:49.187 INFO:teuthology.orchestra.run.smithi176.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T16:21:49.221 INFO:teuthology.orchestra.run.smithi176.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 259 kB/s | 151 kB 00:00 2024-03-25T16:21:49.344 INFO:teuthology.orchestra.run.smithi186.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 826 kB/s | 4.7 MB 00:05 2024-03-25T16:21:49.396 INFO:teuthology.orchestra.run.smithi176.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 981 kB/s | 172 kB 00:00 2024-03-25T16:21:49.621 INFO:teuthology.orchestra.run.smithi176.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-03-25T16:21:49.647 INFO:teuthology.orchestra.run.smithi176.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 95 kB/s | 43 kB 00:00 2024-03-25T16:21:49.731 INFO:teuthology.orchestra.run.smithi176.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-25T16:21:49.992 INFO:teuthology.orchestra.run.smithi176.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-03-25T16:21:49.992 INFO:teuthology.orchestra.run.smithi176.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T16:21:49.992 INFO:teuthology.orchestra.run.smithi176.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 146 kB/s | 19 kB 00:00 2024-03-25T16:21:49.992 INFO:teuthology.orchestra.run.smithi176.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-25T16:21:50.065 INFO:teuthology.orchestra.run.smithi176.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 676 kB/s | 79 kB 00:00 2024-03-25T16:21:50.090 INFO:teuthology.orchestra.run.smithi176.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 252 kB/s | 46 kB 00:00 2024-03-25T16:21:50.199 INFO:teuthology.orchestra.run.smithi176.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T16:21:50.224 INFO:teuthology.orchestra.run.smithi176.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-03-25T16:21:50.317 INFO:teuthology.orchestra.run.smithi176.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 770 kB/s | 90 kB 00:00 2024-03-25T16:21:50.368 INFO:teuthology.orchestra.run.smithi176.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 248 kB/s | 36 kB 00:00 2024-03-25T16:21:50.444 INFO:teuthology.orchestra.run.smithi176.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-25T16:21:50.552 INFO:teuthology.orchestra.run.smithi176.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-25T16:21:50.752 INFO:teuthology.orchestra.run.smithi176.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.1 MB/s | 427 kB 00:00 2024-03-25T16:21:57.094 INFO:teuthology.orchestra.run.smithi176.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 17 MB 00:11 2024-03-25T16:21:57.100 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:57.101 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.3 MB/s | 30 MB 00:12 2024-03-25T16:21:57.125 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:21:57.167 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:21:57.167 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:21:57.582 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:21:57.582 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:21:58.928 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:21:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:21:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:21:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:21:59.135 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:21:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:21:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:21:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:21:59.383 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:21:59.433 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:21:59.504 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:21:59.558 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:21:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:21:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:21:59.700 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:21:59.885 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:21:59.934 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:21:59.964 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:21:59.999 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:22:00.029 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:22:00.105 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:22:00.139 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:00.173 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:22:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:22:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:22:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:22:00.414 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:22:00.487 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:22:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:22:00.747 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:22:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:00.969 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:00.969 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:00.970 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:22:00.970 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:00.970 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:00.970 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:02.403 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:02.429 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:02.870 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:02.893 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:02.894 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:02.894 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:22:02.894 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:02.894 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:02.894 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:03.202 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:22:05.000 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:22:05.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:22:05.002 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:05.002 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:22:05.002 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:22:05.002 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:22:05.002 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:05.003 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:22:05.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:22:05.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:22:05.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:22:05.004 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:22:05.017 INFO:teuthology.orchestra.run.smithi186.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 851 kB/s | 17 MB 00:20 2024-03-25T16:22:05.022 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:05.023 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.3 MB/s | 30 MB 00:22 2024-03-25T16:22:05.047 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:22:05.091 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:22:05.091 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:22:05.524 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:22:05.525 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:22:05.909 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:22:05.909 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:22:05.910 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:22:05.911 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:22:05.912 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:22:05.913 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:22:05.913 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:05.913 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:06.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-03-25T16:22:06.826 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:24 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:06.849 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:22:06.906 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:22:06.925 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:06.949 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:22:07.026 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:22:07.027 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:07.027 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:22:07.027 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:07.053 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:22:07.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-03-25T16:22:07.089 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:22:07.121 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:22:07.212 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:22:07.282 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:22:07.323 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:22:07.394 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:22:07.449 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:22:07.495 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:22:07.544 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:22:07.608 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:22:07.625 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:07.774 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 747 k 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 754 k 2024-03-25T16:22:07.775 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:07.794 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:22:07.842 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:22:07.880 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:22:07.915 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:22:07.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:22:08.021 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:22:08.055 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:08.089 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:22:08.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:22:08.164 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:22:08.199 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:22:08.289 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:22:08.338 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:22:08.411 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:22:08.549 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:22:08.673 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:08.841 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:22:08.868 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:08.894 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:08.894 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:08.895 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:22:08.895 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:08.895 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:08.895 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:10.178 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 311 kB/s | 747 kB 00:02 2024-03-25T16:22:10.179 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:10.179 INFO:teuthology.orchestra.run.smithi176.stdout:Total 311 kB/s | 747 kB 00:02 2024-03-25T16:22:10.179 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:10.185 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:10.185 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:10.191 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:10.192 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:10.267 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:10.344 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:10.371 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:10.371 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:10.371 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:22:10.371 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:10.371 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:10.372 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:10.756 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:10.812 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:10.835 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:10.858 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:11.062 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:11.085 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:11.086 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:11.086 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:22:11.086 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:11.086 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:11.086 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:11.152 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:11.250 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:11.554 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:11.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:12.354 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:30 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:12.513 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:12.514 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 145 k 2024-03-25T16:22:12.515 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 436 k 2024-03-25T16:22:12.516 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:12.977 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:12.977 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:22:12.977 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:22:12.977 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:22:12.978 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:22:12.979 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:22:12.980 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:22:12.980 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:22:12.980 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:12.980 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:22:12.980 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:22:12.981 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:22:12.981 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:22:12.981 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:22:13.371 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 170 kB/s | 145 kB 00:00 2024-03-25T16:22:13.371 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:13.372 INFO:teuthology.orchestra.run.smithi176.stdout:Total 169 kB/s | 145 kB 00:00 2024-03-25T16:22:13.372 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:13.381 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:13.381 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:13.435 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:13.436 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:13.580 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:13.652 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:13.677 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:13.677 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:13.677 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:13.677 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:13.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:22:13.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:13.792 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:22:13.793 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:22:13.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:13.795 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:14.140 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:14.633 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:14.713 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:14.812 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:14.864 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:14.865 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:22:14.865 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:14.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-03-25T16:22:14.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-03-25T16:22:15.463 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:15.502 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:15.562 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:15.614 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:15.615 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:22:15.615 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:15.650 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:22:15.651 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 747 k 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 754 k 2024-03-25T16:22:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:22:15.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:16.252 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:16.401 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:16.402 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-25T16:22:16.403 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-25T16:22:16.404 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:16.404 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.9 M 2024-03-25T16:22:16.404 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-25T16:22:16.404 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:17.280 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 70 kB/s | 31 kB 00:00 2024-03-25T16:22:17.330 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-25T16:22:17.447 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T16:22:17.506 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:22:17.606 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.3 MB/s | 548 kB 00:00 2024-03-25T16:22:17.640 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-25T16:22:17.665 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-25T16:22:17.699 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-03-25T16:22:17.732 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 920 kB/s | 31 kB 00:00 2024-03-25T16:22:17.774 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-03-25T16:22:17.799 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 348 kB/s | 747 kB 00:02 2024-03-25T16:22:17.799 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:17.800 INFO:teuthology.orchestra.run.smithi186.stdout:Total 348 kB/s | 747 kB 00:02 2024-03-25T16:22:17.800 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:22:17.806 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:22:17.807 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:22:17.812 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:22:17.813 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:22:17.816 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-03-25T16:22:17.889 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:22:18.383 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:18.478 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:18.819 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:19.118 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:19.118 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:19.118 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:22:19.119 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:19.119 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:19.119 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:19.337 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:19.921 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:49 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:20.074 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:22:20.075 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:22:20.076 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:20.076 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 145 k 2024-03-25T16:22:20.076 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 436 k 2024-03-25T16:22:20.076 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:22:20.124 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 554 kB/s | 1.8 MB 00:03 2024-03-25T16:22:20.129 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:20.129 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.8 MB/s | 6.9 MB 00:03 2024-03-25T16:22:20.149 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:20.167 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:20.167 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:20.350 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:20.350 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:20.822 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 195 kB/s | 145 kB 00:00 2024-03-25T16:22:20.822 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:20.823 INFO:teuthology.orchestra.run.smithi186.stdout:Total 194 kB/s | 145 kB 00:00 2024-03-25T16:22:20.823 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:22:20.831 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:22:20.831 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:22:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:20.883 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:22:20.883 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:22:20.966 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:21.010 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:21.026 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:22:21.056 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:21.089 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:21.095 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:21.114 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:21.114 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:21.114 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:21.114 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:21.200 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:21.414 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:21.458 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:21.487 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:21.527 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:21.769 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:21.794 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:22.075 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:22.318 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:22.636 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:22.637 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:22.894 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:22.988 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:22.989 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:22.989 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:22.989 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:22.989 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:22.989 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:22.991 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:23.043 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:23.044 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:22:23.044 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:23.116 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:23.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:23.676 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:53 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:23.826 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:23.827 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:23.828 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-25T16:22:23.829 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-03-25T16:22:23.830 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:23.830 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.9 M 2024-03-25T16:22:23.830 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-03-25T16:22:23.830 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:22:23.978 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:23.979 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-25T16:22:23.979 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:23.979 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-25T16:22:23.979 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:23.979 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:23.980 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-25T16:22:23.981 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-25T16:22:23.981 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:23.981 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-03-25T16:22:23.981 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2024-03-25T16:22:23.981 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:24.273 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 283 kB/s | 32 kB 00:00 2024-03-25T16:22:24.331 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 495 kB/s | 17 kB 00:00 2024-03-25T16:22:24.382 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-25T16:22:24.557 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:22:24.682 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-25T16:22:24.732 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T16:22:24.876 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 50 kB/s | 31 kB 00:00 2024-03-25T16:22:25.099 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-03-25T16:22:25.127 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 18 kB/s | 15 kB 00:00 2024-03-25T16:22:25.202 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-03-25T16:22:25.227 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 238 kB 00:00 2024-03-25T16:22:25.250 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-25T16:22:25.383 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-25T16:22:25.408 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-25T16:22:25.525 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T16:22:25.578 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-25T16:22:26.036 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-03-25T16:22:26.103 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 986 kB/s | 1.8 MB 00:01 2024-03-25T16:22:26.395 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 469 kB/s | 548 kB 00:01 2024-03-25T16:22:26.504 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T16:22:26.771 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2024-03-25T16:22:26.821 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 201 kB/s | 144 kB 00:00 2024-03-25T16:22:28.646 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 799 kB/s | 2.0 MB 00:02 2024-03-25T16:22:28.652 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:28.653 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.4 MB/s | 6.9 MB 00:04 2024-03-25T16:22:28.672 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:22:28.690 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:22:28.690 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:22:28.868 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:22:28.868 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:22:29.385 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:22:29.463 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:29.498 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:29.545 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:29.575 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:29.678 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:29.800 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:29.888 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:29.939 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:29.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:29.999 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:30.241 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:30.274 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:31.106 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:31.107 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:31.455 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:31.456 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:31.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:32.284 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:32.433 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-25T16:22:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:22:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-25T16:22:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:32.437 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:32.437 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:22:32.437 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-25T16:22:32.437 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-03-25T16:22:32.437 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:32.438 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 37 M 2024-03-25T16:22:32.438 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-03-25T16:22:32.438 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:22:32.903 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 214 kB/s | 32 kB 00:00 2024-03-25T16:22:32.928 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 695 kB/s | 17 kB 00:00 2024-03-25T16:22:32.953 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-25T16:22:33.162 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-25T16:22:33.312 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T16:22:33.354 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T16:22:33.413 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2024-03-25T16:22:33.714 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 730 kB/s | 7.4 MB 00:10 2024-03-25T16:22:34.535 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:34.536 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.5 MB/s | 37 MB 00:10 2024-03-25T16:22:34.572 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 228 kB/s | 195 kB 00:00 2024-03-25T16:22:34.580 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:34.788 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T16:22:34.961 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:34.961 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:35.147 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 464 kB/s | 803 kB 00:01 2024-03-25T16:22:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:35.756 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:22:35.787 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:22:35.880 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:35.915 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:22:36.092 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:36.122 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:36.198 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:22:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:22:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:22:38.233 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:22:39.264 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:22:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:41.807 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:22:41.808 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:22:41.808 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:22:41.808 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:22:41.808 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:22:42.240 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:42.241 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:42.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:22:43.283 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:43.445 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:22:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-03-25T16:22:43.447 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:43.448 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-25T16:22:43.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-25T16:22:43.448 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:43.779 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-25T16:22:43.804 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-03-25T16:22:43.854 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:22:43.880 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:22:43.905 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T16:22:43.930 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-25T16:22:44.294 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 41 kB/s | 15 kB 00:00 2024-03-25T16:22:44.314 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 77 kB/s | 52 kB 00:00 2024-03-25T16:22:44.331 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 76 kB/s | 32 kB 00:00 2024-03-25T16:22:44.623 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 616 kB/s | 203 kB 00:00 2024-03-25T16:22:44.749 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 715 kB/s | 90 kB 00:00 2024-03-25T16:22:44.774 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 134 kB/s | 59 kB 00:00 2024-03-25T16:22:44.807 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-25T16:22:44.812 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:44.812 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T16:22:44.812 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:44.824 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:44.824 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:44.915 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:44.916 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:45.431 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:45.598 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:22:45.650 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:22:45.705 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:22:45.742 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:22:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:22:45.877 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:22:45.994 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:22:45.996 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 571 kB/s | 7.4 MB 00:13 2024-03-25T16:22:46.000 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:46.000 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.7 MB/s | 37 MB 00:13 2024-03-25T16:22:46.043 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:22:46.051 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:22:46.069 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:22:46.069 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:22:46.454 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:22:46.455 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:22:46.780 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:22:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:22:46.858 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:22:46.906 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:22:46.928 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:47.174 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:22:47.244 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:22:47.275 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:22:47.380 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:47.412 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:22:47.591 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:47.619 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:47.703 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:22:47.732 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:47.732 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:22:47.732 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:22:47.733 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:22:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:22:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:22:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:22:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:48.095 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:48.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:22:48.369 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:22:48.484 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:22:48.932 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:49.082 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:49.082 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:49.082 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:22:49.083 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-25T16:22:49.084 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:49.084 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2024-03-25T16:22:49.084 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2024-03-25T16:22:49.084 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:49.509 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:22:49.736 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:22:50.051 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:22:50.134 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 694 kB/s | 58 kB 00:00 2024-03-25T16:22:50.226 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-03-25T16:22:50.251 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 241 kB/s | 248 kB 00:01 2024-03-25T16:22:50.527 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 106 kB/s | 138 kB 00:01 2024-03-25T16:22:50.530 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:50.530 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-25T16:22:50.543 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:50.555 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:50.555 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:50.645 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:50.646 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:50.770 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:22:50.803 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:50.950 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:51.122 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:22:51.177 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:22:51.684 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:22:51.771 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:22:51.833 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:22:51.866 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:22:52.656 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:22:52.656 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:22:52.656 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:22:52.656 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:22:52.656 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:22:52.657 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:22:52.951 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:22:52.952 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:22:52.952 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:22:52.952 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:52.952 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:53.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-03-25T16:22:53.366 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:22:53.367 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:22:53.763 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:11 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:22:53.821 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:53.822 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:22:53.913 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:53.913 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-25T16:22:53.914 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:53.915 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 964 k 2024-03-25T16:22:53.915 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2024-03-25T16:22:53.915 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:22:54.036 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 647 kB/s | 79 kB 00:00 2024-03-25T16:22:54.242 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:22:54.813 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:24 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:22:54.964 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:22:54.965 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:54.966 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-03-25T16:22:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:22:55.155 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 233 kB/s | 24 kB 00:00 2024-03-25T16:22:55.205 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-25T16:22:55.255 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:22:55.280 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:22:55.306 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:22:55.331 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 870 kB/s | 43 kB 00:00 2024-03-25T16:22:55.506 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-25T16:22:55.740 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-google-auth-2.28.2-1.el9.noarch 868 kB/s | 203 kB 00:00 2024-03-25T16:22:55.765 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 73 kB/s | 52 kB 00:00 2024-03-25T16:22:55.941 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-03-25T16:22:56.008 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T16:22:57.520 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 246 kB/s | 885 kB 00:03 2024-03-25T16:22:57.521 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:57.522 INFO:teuthology.orchestra.run.smithi176.stdout:Total 267 kB/s | 964 kB 00:03 2024-03-25T16:22:57.522 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:22:57.545 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:22:57.545 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:22:57.621 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:22:57.621 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:22:57.812 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:22:57.954 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:22:58.018 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:22:58.901 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:22:58.902 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:58.902 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:22:58.902 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:22:58.902 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:58.902 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:22:59.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-volume 2024-03-25T16:22:59.761 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:17 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:22:59.918 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:22:59.919 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 332 k 2024-03-25T16:22:59.920 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.5 M 2024-03-25T16:22:59.920 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:00.024 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 660 kB/s | 69 kB 00:00 2024-03-25T16:23:00.833 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 289 kB/s | 264 kB 00:00 2024-03-25T16:23:00.834 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:00.834 INFO:teuthology.orchestra.run.smithi176.stdout:Total 363 kB/s | 332 kB 00:00 2024-03-25T16:23:00.835 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:00.844 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:00.844 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:00.888 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:00.888 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:01.034 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:01.214 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:23:01.260 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:01.276 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:01.277 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:01.277 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:23:01.277 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:02.047 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:23:02.386 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:02.386 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:02.649 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-cachetools-4.2.4-1.el9.noarch. 4.4 kB/s | 32 kB 00:07 2024-03-25T16:23:02.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-03-25T16:23:03.350 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:03.499 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:03.500 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:23:03.501 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:03.501 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 127 k 2024-03-25T16:23:03.501 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 456 k 2024-03-25T16:23:03.501 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:04.355 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 148 kB/s | 127 kB 00:00 2024-03-25T16:23:04.356 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:04.356 INFO:teuthology.orchestra.run.smithi176.stdout:Total 148 kB/s | 127 kB 00:00 2024-03-25T16:23:04.356 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:04.363 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:04.363 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:04.423 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:04.424 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:04.688 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:05.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:05.465 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:05.465 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:05.465 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:05.466 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:05.466 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:05.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-03-25T16:23:06.262 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:24 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:06.361 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:06.415 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:06.416 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:06.416 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:06.487 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-03-25T16:23:07.058 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:07.208 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 k 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 140 k 2024-03-25T16:23:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:07.660 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 69 kB/s | 31 kB 00:00 2024-03-25T16:23:07.660 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:07.660 INFO:teuthology.orchestra.run.smithi176.stdout:Total 69 kB/s | 31 kB 00:00 2024-03-25T16:23:07.661 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:07.664 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:07.664 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:07.699 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:07.699 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:07.820 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:07.926 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:08.319 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:08.581 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:08.744 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-03-25T16:23:09.325 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:09.434 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:09.487 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:09.488 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:09.488 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:09.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-03-25T16:23:10.108 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:28 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:10.206 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:10.258 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:10.259 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:10.259 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:10.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-03-25T16:23:10.885 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:28 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:10.983 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:11.035 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:11.037 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:11.037 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:11.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-03-25T16:23:11.655 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:11.754 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:11.807 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:11.808 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:11.808 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:11.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-03-25T16:23:12.438 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:12.541 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:12.596 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:12.597 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:12.597 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:12.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-03-25T16:23:13.262 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:31 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:13.363 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:13.417 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:13.418 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T16:23:13.418 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:13.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-03-25T16:23:14.061 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:14.210 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:14.211 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:23:14.212 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:14.212 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2024-03-25T16:23:14.212 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 231 k 2024-03-25T16:23:14.212 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:14.905 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 123 kB/s | 85 kB 00:00 2024-03-25T16:23:14.905 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:14.905 INFO:teuthology.orchestra.run.smithi176.stdout:Total 123 kB/s | 85 kB 00:00 2024-03-25T16:23:14.906 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:14.969 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:14.969 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:15.103 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:15.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.600 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.889 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.889 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:15.889 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:15.889 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.889 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:15.890 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:16.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-03-25T16:23:16.684 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:34 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:16.837 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:16.838 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.1 M 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 13 M 2024-03-25T16:23:16.839 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:20.016 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 995 kB/s | 3.1 MB 00:03 2024-03-25T16:23:20.016 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:20.017 INFO:teuthology.orchestra.run.smithi176.stdout:Total 995 kB/s | 3.1 MB 00:03 2024-03-25T16:23:20.017 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:20.028 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:20.028 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:20.104 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:20.105 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:20.641 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:20.641 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:23:20.641 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:20.641 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:21.756 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:21.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-03-25T16:23:22.570 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:40 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:22.720 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:22.720 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 171 k 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 494 k 2024-03-25T16:23:22.721 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:23.544 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 209 kB/s | 171 kB 00:00 2024-03-25T16:23:23.544 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:23.545 INFO:teuthology.orchestra.run.smithi176.stdout:Total 208 kB/s | 171 kB 00:00 2024-03-25T16:23:23.545 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:23:23.553 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:23:23.553 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:23:23.613 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:23:23.614 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:23:23.758 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:23:23.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:24.303 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:24.612 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:23:24.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:23:25.431 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:23:25.584 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:23:25.586 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:25.586 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:23:25.586 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:25.586 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:23:25.587 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:23:25.588 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-25T16:23:25.588 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:23:25.588 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 91 M 2024-03-25T16:23:25.588 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 498 M 2024-03-25T16:23:25.588 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:23:27.562 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 kB/s | 59 kB 00:31 2024-03-25T16:23:27.565 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:27.565 INFO:teuthology.orchestra.run.smithi186.stdout:Total 64 kB/s | 2.0 MB 00:32 2024-03-25T16:23:27.565 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:27.581 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:27.581 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:27.679 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:27.679 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:28.194 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:28.362 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:23:28.413 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:23:28.469 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:23:28.506 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:23:28.547 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:23:28.642 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:23:28.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:23:28.799 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:23:29.518 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:29.571 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:29.605 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:29.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:29.675 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:30.440 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:30.440 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:23:30.440 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:23:30.440 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:23:30.440 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:23:30.441 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:23:30.780 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:23:30.781 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:30.782 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:31.067 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:23:31.632 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:31.782 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:23:31.783 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-03-25T16:23:31.784 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:31.785 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.8 M 2024-03-25T16:23:31.785 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-03-25T16:23:31.785 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:32.373 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-25T16:23:32.540 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:23:32.716 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2024-03-25T16:23:32.791 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:23:32.816 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-03-25T16:23:32.966 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 184 kB/s | 138 kB 00:00 2024-03-25T16:23:32.968 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:32.968 INFO:teuthology.orchestra.run.smithi186.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-25T16:23:32.981 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:32.991 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:32.991 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:33.079 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:33.082 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:33.376 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:33.557 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:33.614 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:34.135 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:34.220 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:34.282 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:34.305 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:35.132 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:35.437 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:35.438 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:35.438 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:35.680 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-03-25T16:23:36.253 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:06 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:36.404 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:36.404 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:36.404 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:23:36.404 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:36.404 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-03-25T16:23:36.405 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:36.406 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 964 k 2024-03-25T16:23:36.406 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.8 M 2024-03-25T16:23:36.406 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:36.518 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 702 kB/s | 79 kB 00:00 2024-03-25T16:23:39.093 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 329 kB/s | 885 kB 00:02 2024-03-25T16:23:39.094 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:39.095 INFO:teuthology.orchestra.run.smithi186.stdout:Total 358 kB/s | 964 kB 00:02 2024-03-25T16:23:39.095 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:39.118 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:39.118 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:39.196 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:39.196 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:39.385 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:39.527 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:23:39.592 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:40.338 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:40.338 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:40.665 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:40.942 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-volume 2024-03-25T16:23:41.527 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:41.683 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:41.683 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 332 k 2024-03-25T16:23:41.684 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.5 M 2024-03-25T16:23:41.685 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:41.825 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 490 kB/s | 69 kB 00:00 2024-03-25T16:23:42.550 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 305 kB/s | 264 kB 00:00 2024-03-25T16:23:42.551 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:42.552 INFO:teuthology.orchestra.run.smithi186.stdout:Total 383 kB/s | 332 kB 00:00 2024-03-25T16:23:42.552 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:42.561 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:42.561 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:42.603 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:42.604 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:42.755 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:42.941 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:23:42.977 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:42.994 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:42.994 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:42.994 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:23:42.994 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:43.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:44.150 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:44.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-03-25T16:23:45.081 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:45.230 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:45.230 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:23:45.231 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:45.232 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 127 k 2024-03-25T16:23:45.232 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 456 k 2024-03-25T16:23:45.232 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:45.840 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 208 kB/s | 127 kB 00:00 2024-03-25T16:23:45.840 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:45.841 INFO:teuthology.orchestra.run.smithi186.stdout:Total 208 kB/s | 127 kB 00:00 2024-03-25T16:23:45.841 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:45.846 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:45.846 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:45.903 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:45.903 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:46.065 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:46.156 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:46.604 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:46.905 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:47.192 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-03-25T16:23:47.767 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:47.867 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:47.920 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:47.922 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:47.922 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:47.999 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-03-25T16:23:48.567 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:48.716 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:48.717 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 31 k 2024-03-25T16:23:48.718 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 140 k 2024-03-25T16:23:48.718 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:49.177 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:23:49.178 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:49.178 INFO:teuthology.orchestra.run.smithi186.stdout:Total 68 kB/s | 31 kB 00:00 2024-03-25T16:23:49.178 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:49.182 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:49.182 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:49.212 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:49.212 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:49.349 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:49.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:49.906 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:50.202 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:50.202 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:50.202 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:50.202 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:50.203 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:50.203 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:50.376 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-03-25T16:23:50.942 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:51.039 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:51.090 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:51.091 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:51.091 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:51.141 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-03-25T16:23:51.711 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:51.804 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:51.856 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:51.856 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:51.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-03-25T16:23:52.470 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:52.567 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:52.618 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:52.619 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:52.619 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:52.668 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-03-25T16:23:53.235 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:53.331 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:53.382 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:53.383 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:53.383 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:53.442 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-03-25T16:23:54.008 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:54.105 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:54.158 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:54.159 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:54.159 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:54.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-03-25T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:54.873 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T16:23:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:54.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-03-25T16:23:55.235 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 775 kB/s | 22 MB 00:29 2024-03-25T16:23:55.545 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:25 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:55.695 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:55.696 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:23:55.697 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:55.697 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 85 k 2024-03-25T16:23:55.697 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 231 k 2024-03-25T16:23:55.697 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:23:56.239 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 157 kB/s | 85 kB 00:00 2024-03-25T16:23:56.239 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:56.239 INFO:teuthology.orchestra.run.smithi186.stdout:Total 157 kB/s | 85 kB 00:00 2024-03-25T16:23:56.240 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:23:56.247 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:23:56.247 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:23:56.302 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:23:56.303 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:23:56.439 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:23:56.546 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:56.943 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:57.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:57.235 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:57.235 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:23:57.235 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:57.235 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:57.235 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:23:57.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-03-25T16:23:58.050 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:23:58.202 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:23:58.203 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:58.203 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:23:58.203 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:58.203 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:23:58.203 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.1 M 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 13 M 2024-03-25T16:23:58.204 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:24:02.733 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 698 kB/s | 3.1 MB 00:04 2024-03-25T16:24:02.734 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:02.734 INFO:teuthology.orchestra.run.smithi186.stdout:Total 698 kB/s | 3.1 MB 00:04 2024-03-25T16:24:02.735 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:24:02.746 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:24:02.746 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:24:02.822 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:24:02.823 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:24:03.250 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:24:03.317 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:24:03.342 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:04.469 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:04.469 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:04.469 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:24:04.469 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:04.470 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:04.470 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:24:04.728 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-03-25T16:24:05.311 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:24:05.469 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:24:05.470 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 171 k 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 494 k 2024-03-25T16:24:05.471 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:24:06.530 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 162 kB/s | 171 kB 00:01 2024-03-25T16:24:06.531 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:06.532 INFO:teuthology.orchestra.run.smithi186.stdout:Total 162 kB/s | 171 kB 00:01 2024-03-25T16:24:06.532 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:24:06.539 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:24:06.539 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:24:06.594 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:24:06.594 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:24:06.743 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:24:06.836 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:07.312 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:07.593 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:24:07.593 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:07.594 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:24:07.594 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:07.594 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:07.594 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:24:07.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:24:08.412 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:24:08.568 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-03-25T16:24:08.569 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:24:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 91 M 2024-03-25T16:24:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 498 M 2024-03-25T16:24:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:25:04.678 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 410 kB/s | 22 MB 00:56 2024-03-25T16:25:35.796 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 536 kB/s | 68 MB 02:10 2024-03-25T16:25:35.797 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:35.797 INFO:teuthology.orchestra.run.smithi176.stdout:Total 712 kB/s | 91 MB 02:10 2024-03-25T16:25:35.797 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:25:35.802 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:25:35.802 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:25:36.394 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:25:36.394 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:25:39.515 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:25:46.638 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:46.721 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:49.921 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:49.921 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:25:50.187 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:25:50.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:25:50.972 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:25:51.121 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:25:51.122 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:25:51.122 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:25:51.122 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 416 M 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.7 G 2024-03-25T16:25:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:27:13.953 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 376 kB/s | 68 MB 03:05 2024-03-25T16:27:13.954 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:13.954 INFO:teuthology.orchestra.run.smithi186.stdout:Total 500 kB/s | 91 MB 03:05 2024-03-25T16:27:13.955 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:27:13.958 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:27:13.959 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:27:14.543 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:27:14.543 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:27:17.660 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:27:24.736 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:24.803 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:28.125 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:27:28.125 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:27:28.396 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:27:28.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:27:29.197 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:07:59 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:27:29.348 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:27:29.349 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 416 M 2024-03-25T16:27:29.350 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.7 G 2024-03-25T16:27:29.350 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:54:10.707 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 251 kB/s | 416 MB 28:19 2024-03-25T16:54:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:54:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:Total 251 kB/s | 416 MB 28:19 2024-03-25T16:54:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:54:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:54:10.727 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:54:13.176 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:54:13.177 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:54:55.258 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:54:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:02.115 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:02.390 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:02.390 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:02.390 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:55:02.390 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:02.390 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:02.391 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:55:02.623 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:55:03.190 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:35:21 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:55:03.339 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:55:03.340 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:03.341 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.4 M 2024-03-25T16:55:03.341 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 6.4 M 2024-03-25T16:55:03.341 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:55:06.079 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 534 kB/s | 1.4 MB 00:02 2024-03-25T16:55:06.080 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:06.080 INFO:teuthology.orchestra.run.smithi176.stdout:Total 533 kB/s | 1.4 MB 00:02 2024-03-25T16:55:06.080 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T16:55:06.084 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T16:55:06.084 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T16:55:06.128 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T16:55:06.129 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T16:55:06.397 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T16:55:06.494 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:06.927 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:07.199 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:07.200 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:07.200 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T16:55:07.200 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:07.200 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:07.200 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T16:55:07.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:55:07.958 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:35:25 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T16:55:08.109 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T16:55:08.110 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:55:08.110 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T16:55:08.110 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:55:08.110 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 176 M 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 753 M 2024-03-25T16:55:08.111 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T16:56:53.538 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 242 kB/s | 416 MB 29:24 2024-03-25T16:56:53.538 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:53.539 INFO:teuthology.orchestra.run.smithi186.stdout:Total 242 kB/s | 416 MB 29:24 2024-03-25T16:56:53.539 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:56:53.543 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:56:53.543 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:56:56.011 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:56:56.011 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:57:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:57:38.859 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:57:46.535 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:57:46.814 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:57:46.815 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:46.815 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:57:46.815 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:46.815 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:46.815 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:57:47.091 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:57:47.682 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:38:17 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:57:47.849 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:57:47.850 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:57:47.850 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.4 M 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 6.4 M 2024-03-25T16:57:47.851 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T16:57:51.478 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 403 kB/s | 1.4 MB 00:03 2024-03-25T16:57:51.479 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:51.479 INFO:teuthology.orchestra.run.smithi186.stdout:Total 403 kB/s | 1.4 MB 00:03 2024-03-25T16:57:51.479 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T16:57:51.484 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T16:57:51.484 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T16:57:51.528 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T16:57:51.528 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T16:57:51.798 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T16:57:51.909 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:52.293 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:52.572 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:57:52.572 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:52.572 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T16:57:52.572 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:52.573 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:52.573 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T16:57:52.766 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:57:53.333 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:38:23 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T16:57:53.481 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T16:57:53.482 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:57:53.482 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T16:57:53.482 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 176 M 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 753 M 2024-03-25T16:57:53.483 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T17:02:44.443 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 394 kB/s | 176 MB 07:36 2024-03-25T17:02:44.444 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:02:44.444 INFO:teuthology.orchestra.run.smithi176.stdout:Total 394 kB/s | 176 MB 07:36 2024-03-25T17:02:44.444 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T17:02:44.447 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T17:02:44.447 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T17:02:45.510 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T17:02:45.511 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T17:03:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T17:03:03.580 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:08.117 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:08.386 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:08.386 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:03:08.387 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T17:03:08.387 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:03:08.387 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:03:08.387 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:03:08.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:03:09.160 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:43:27 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:03:09.310 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:03:09.311 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:03:09.311 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 672 M 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 3.1 G 2024-03-25T17:03:09.312 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T17:09:08.965 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 266 kB/s | 176 MB 11:15 2024-03-25T17:09:08.965 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:08.966 INFO:teuthology.orchestra.run.smithi186.stdout:Total 266 kB/s | 176 MB 11:15 2024-03-25T17:09:08.966 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T17:09:08.969 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T17:09:08.969 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T17:09:10.028 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T17:09:10.028 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T17:09:28.145 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T17:09:28.235 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:32.734 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:09:33.011 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:09:33.214 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:09:33.779 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:50:03 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:09:33.928 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:09:33.928 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:09:33.928 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T17:09:33.928 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 672 M 2024-03-25T17:09:33.929 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 3.1 G 2024-03-25T17:09:33.930 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T17:37:59.843 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 329 kB/s | 672 MB 34:50 2024-03-25T17:37:59.867 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:59.867 INFO:teuthology.orchestra.run.smithi176.stdout:Total 329 kB/s | 672 MB 34:50 2024-03-25T17:37:59.867 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T17:37:59.868 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T17:37:59.868 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T17:38:03.804 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T17:38:03.804 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T17:39:14.544 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T17:39:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:23.177 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:39:23.419 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:39:23.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:39:24.273 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:19:42 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:39:24.369 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:39:24.421 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:39:24.422 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-25T17:39:24.422 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:39:24.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:39:25.056 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:19:43 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:39:25.203 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:39:25.204 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 20 M 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 103 M 2024-03-25T17:39:25.205 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T17:40:32.910 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 298 kB/s | 20 MB 01:07 2024-03-25T17:40:32.911 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:32.911 INFO:teuthology.orchestra.run.smithi176.stdout:Total 298 kB/s | 20 MB 01:07 2024-03-25T17:40:32.911 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T17:40:32.915 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T17:40:32.915 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T17:40:33.065 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T17:40:33.065 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T17:40:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T17:40:35.465 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:36.370 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:40:36.641 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:40:36.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:40:37.411 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:20:55 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:40:37.559 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T17:40:37.560 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:40:37.561 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T17:40:37.561 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:40:37.561 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 48 M 2024-03-25T17:40:37.561 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 173 M 2024-03-25T17:40:37.561 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T17:43:24.348 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 293 kB/s | 48 MB 02:46 2024-03-25T17:43:24.348 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:24.349 INFO:teuthology.orchestra.run.smithi176.stdout:Total 293 kB/s | 48 MB 02:46 2024-03-25T17:43:24.349 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T17:43:24.352 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T17:43:24.352 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T17:43:24.664 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T17:43:24.664 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T17:43:29.409 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T17:43:29.561 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:30.881 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:31.136 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:31.136 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:43:31.137 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T17:43:31.137 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:31.137 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:43:31.137 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:43:31.317 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:43:31.887 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:23:49 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:43:32.037 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:43:32.037 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 99 M 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 449 M 2024-03-25T17:43:32.038 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T17:48:02.087 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 374 kB/s | 99 MB 04:30 2024-03-25T17:48:02.087 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:02.088 INFO:teuthology.orchestra.run.smithi176.stdout:Total 374 kB/s | 99 MB 04:30 2024-03-25T17:48:02.088 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T17:48:02.090 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T17:48:02.090 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T17:48:02.699 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T17:48:02.699 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T17:48:12.988 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T17:48:13.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:15.866 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:16.115 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:16.116 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:48:16.116 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T17:48:16.116 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:16.116 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:48:16.116 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T17:48:16.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:48:16.868 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:28:34 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T17:48:17.018 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 287 M 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.2 G 2024-03-25T17:48:17.019 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T17:49:30.273 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 287 kB/s | 672 MB 39:56 2024-03-25T17:49:30.274 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:30.274 INFO:teuthology.orchestra.run.smithi186.stdout:Total 287 kB/s | 672 MB 39:56 2024-03-25T17:49:30.274 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T17:49:30.278 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T17:49:30.278 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T17:49:34.257 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T17:49:34.257 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T17:50:45.525 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T17:50:45.706 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:55.041 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:50:55.322 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:50:55.628 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:50:56.198 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:31:26 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:50:56.294 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:56.346 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:50:56.347 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-25T17:50:56.347 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:50:56.418 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:50:56.984 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:31:26 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:50:57.135 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:50:57.136 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 20 M 2024-03-25T17:50:57.137 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 103 M 2024-03-25T17:50:57.137 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T17:51:31.742 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 583 kB/s | 20 MB 00:34 2024-03-25T17:51:31.743 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:31.743 INFO:teuthology.orchestra.run.smithi186.stdout:Total 583 kB/s | 20 MB 00:34 2024-03-25T17:51:31.743 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T17:51:31.746 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T17:51:31.746 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T17:51:31.894 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T17:51:31.902 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T17:51:34.131 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T17:51:34.323 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:35.333 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:35.597 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:35.597 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:51:35.598 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T17:51:35.598 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:35.598 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:51:35.598 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:51:35.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:51:36.355 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:32:06 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:51:36.503 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:51:36.504 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:51:36.504 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T17:51:36.504 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:51:36.504 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 48 M 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 173 M 2024-03-25T17:51:36.505 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T17:53:57.044 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 348 kB/s | 48 MB 02:20 2024-03-25T17:53:57.044 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:57.044 INFO:teuthology.orchestra.run.smithi186.stdout:Total 348 kB/s | 48 MB 02:20 2024-03-25T17:53:57.045 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T17:53:57.047 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T17:53:57.047 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T17:53:57.364 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T17:53:57.364 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T17:54:02.137 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T17:54:02.249 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:03.570 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:54:03.847 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:54:04.041 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:54:04.607 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:34:34 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:54:04.755 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:54:04.756 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T17:54:04.757 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:54:04.757 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 99 M 2024-03-25T17:54:04.757 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 449 M 2024-03-25T17:54:04.757 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T17:58:18.223 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 398 kB/s | 99 MB 04:13 2024-03-25T17:58:18.223 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:18.223 INFO:teuthology.orchestra.run.smithi186.stdout:Total 398 kB/s | 99 MB 04:13 2024-03-25T17:58:18.223 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T17:58:18.226 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T17:58:18.226 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T17:58:18.835 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T17:58:18.836 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T17:58:29.116 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T17:58:29.219 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:32.164 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:32.445 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:32.445 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:58:32.446 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T17:58:32.446 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:32.446 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:58:32.446 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T17:58:32.641 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:58:33.208 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:39:03 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T17:58:33.356 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T17:58:33.357 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:58:33.357 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T17:58:33.357 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:58:33.357 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 287 M 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.2 G 2024-03-25T17:58:33.358 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:01:59.155 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 357 kB/s | 287 MB 13:42 2024-03-25T18:01:59.155 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:59.155 INFO:teuthology.orchestra.run.smithi176.stdout:Total 357 kB/s | 287 MB 13:42 2024-03-25T18:01:59.155 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:01:59.158 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:01:59.158 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:02:00.866 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:02:00.866 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:02:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:02:30.298 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:37.596 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:37.878 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:02:38.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:02:38.663 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:42:56 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:02:38.813 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:02:38.814 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:38.814 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 5.9 M 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 28 M 2024-03-25T18:02:38.815 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:02:57.233 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 330 kB/s | 5.9 MB 00:18 2024-03-25T18:02:57.234 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:57.234 INFO:teuthology.orchestra.run.smithi176.stdout:Total 330 kB/s | 5.9 MB 00:18 2024-03-25T18:02:57.235 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:02:57.239 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:02:57.239 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:02:57.310 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:02:57.311 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:02:58.045 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:02:58.157 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:58.962 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:58.962 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:58.962 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:02:58.962 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:58.963 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:58.963 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:02:59.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:02:59.724 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:43:17 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:02:59.873 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:02:59.874 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:02:59.875 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 109 k 2024-03-25T18:02:59.875 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 440 k 2024-03-25T18:02:59.875 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:03:00.686 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 134 kB/s | 109 kB 00:00 2024-03-25T18:03:00.687 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:00.687 INFO:teuthology.orchestra.run.smithi176.stdout:Total 134 kB/s | 109 kB 00:00 2024-03-25T18:03:00.687 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:03:00.690 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:03:00.690 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:03:00.725 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:03:00.726 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:03:00.843 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:03:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.340 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.595 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.595 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:01.595 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:03:01.595 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:01.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:01.596 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:03:01.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:03:02.338 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:43:20 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:03:02.487 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:03:02.488 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:02.489 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:03:02.489 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:02.489 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 5.4 M 2024-03-25T18:03:02.489 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 25 M 2024-03-25T18:03:02.489 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:03:24.170 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 253 kB/s | 5.4 MB 00:21 2024-03-25T18:03:24.170 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:24.171 INFO:teuthology.orchestra.run.smithi176.stdout:Total 253 kB/s | 5.4 MB 00:21 2024-03-25T18:03:24.171 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:03:24.175 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:03:24.175 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:03:24.243 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:03:24.243 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:03:24.907 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:03:25.010 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:03:25.524 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:25.782 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:03:25.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2-debuginfo 2024-03-25T18:03:26.550 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:43:44 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:03:26.698 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:03:26.699 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:03:26.700 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:03:26.700 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:03:26.700 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 44 M 2024-03-25T18:03:26.700 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 253 M 2024-03-25T18:03:26.700 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:05:41.905 INFO:teuthology.orchestra.run.smithi176.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 335 kB/s | 44 MB 02:15 2024-03-25T18:05:41.906 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:41.906 INFO:teuthology.orchestra.run.smithi176.stdout:Total 335 kB/s | 44 MB 02:15 2024-03-25T18:05:41.906 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:05:41.909 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:05:41.909 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:05:42.208 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:05:42.208 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:05:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:05:47.377 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:05:49.402 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:05:49.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:05:50.171 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:46:08 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:05:50.321 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:05:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:05:50.323 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 52 M 2024-03-25T18:05:50.323 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 308 M 2024-03-25T18:05:50.323 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:07:49.866 INFO:teuthology.orchestra.run.smithi176.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 444 kB/s | 52 MB 01:59 2024-03-25T18:07:49.866 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:49.866 INFO:teuthology.orchestra.run.smithi176.stdout:Total 444 kB/s | 52 MB 01:59 2024-03-25T18:07:49.866 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:07:49.869 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:07:49.869 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:07:50.212 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:07:50.212 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:07:56.189 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:07:56.396 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:58.837 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:58.837 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:07:58.838 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:07:58.838 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:58.838 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:07:58.838 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:07:59.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:07:59.595 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:17 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:07:59.745 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:07:59.746 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 615 k 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.1 M 2024-03-25T18:07:59.747 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:08:02.358 INFO:teuthology.orchestra.run.smithi176.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 235 kB/s | 615 kB 00:02 2024-03-25T18:08:02.359 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:02.359 INFO:teuthology.orchestra.run.smithi176.stdout:Total 235 kB/s | 615 kB 00:02 2024-03-25T18:08:02.359 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:08:02.363 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:08:02.363 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:08:02.404 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:08:02.405 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:08:02.570 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:08:02.666 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:03.348 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:08:03.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:08:04.102 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:22 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:08:04.253 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:08:04.254 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:04.254 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:08:04.254 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 771 k 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.7 M 2024-03-25T18:08:04.255 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:08:07.627 INFO:teuthology.orchestra.run.smithi176.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 229 kB/s | 771 kB 00:03 2024-03-25T18:08:07.628 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:07.628 INFO:teuthology.orchestra.run.smithi176.stdout:Total 229 kB/s | 771 kB 00:03 2024-03-25T18:08:07.629 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:08:07.633 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:08:07.633 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:08:07.675 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:08:07.675 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:08:07.859 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:08:07.959 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:08.315 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:08.562 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:08:08.562 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:08.563 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:08:08.563 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:08.563 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:08.563 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:08:08.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:08:09.346 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:27 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:08:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.1 M 2024-03-25T18:08:09.496 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 4.3 M 2024-03-25T18:08:09.497 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:08:16.798 INFO:teuthology.orchestra.run.smithi176.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 158 kB/s | 1.1 MB 00:07 2024-03-25T18:08:16.799 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:16.799 INFO:teuthology.orchestra.run.smithi176.stdout:Total 158 kB/s | 1.1 MB 00:07 2024-03-25T18:08:16.800 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:08:16.804 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:08:16.804 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:08:16.848 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:08:16.848 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:08:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:08:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:17.844 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:08:18.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:08:18.633 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:36 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:08:18.784 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:08:18.785 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 337 k 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.0 M 2024-03-25T18:08:18.786 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:08:21.193 INFO:teuthology.orchestra.run.smithi176.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 140 kB/s | 337 kB 00:02 2024-03-25T18:08:21.193 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:21.194 INFO:teuthology.orchestra.run.smithi176.stdout:Total 140 kB/s | 337 kB 00:02 2024-03-25T18:08:21.194 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:08:21.198 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:08:21.198 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:08:21.238 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:08:21.238 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:08:21.382 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:08:21.492 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:21.849 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:08:22.102 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:22.103 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:08:22.103 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:22.103 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:22.103 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:08:22.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:08:22.891 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:40 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:08:23.041 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:08:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:23.043 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:08:23.043 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:23.043 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 509 k 2024-03-25T18:08:23.043 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.5 M 2024-03-25T18:08:23.043 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:08:25.904 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 178 kB/s | 509 kB 00:02 2024-03-25T18:08:25.904 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:25.905 INFO:teuthology.orchestra.run.smithi176.stdout:Total 178 kB/s | 509 kB 00:02 2024-03-25T18:08:25.905 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:08:25.909 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:08:25.909 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:08:25.948 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:08:25.949 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:08:26.136 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:08:26.245 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:26.614 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:26.868 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:26.869 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:26.869 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:08:26.869 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:26.869 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:26.869 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:08:27.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:08:27.667 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:48:45 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:08:27.816 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:08:27.816 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:27.816 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:08:27.816 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:27.816 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 52 M 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 298 M 2024-03-25T18:08:27.817 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:10:09.601 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 519 kB/s | 52 MB 01:41 2024-03-25T18:10:09.602 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:09.602 INFO:teuthology.orchestra.run.smithi176.stdout:Total 519 kB/s | 52 MB 01:41 2024-03-25T18:10:09.602 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:10:09.605 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:10:09.605 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:10:09.938 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:10:09.939 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:10:15.821 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:10:15.915 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:17.968 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:18.225 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:10:18.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:10:18.993 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 1:50:36 ago on Mon 25 Mar 2024 04:19:42 PM UTC. 2024-03-25T18:10:19.141 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-25T18:10:19.142 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-25T18:10:19.143 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:19.143 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.3 M 2024-03-25T18:10:19.143 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 5.5 M 2024-03-25T18:10:19.143 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-25T18:10:22.211 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 418 kB/s | 1.3 MB 00:03 2024-03-25T18:10:22.212 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:22.213 INFO:teuthology.orchestra.run.smithi176.stdout:Total 418 kB/s | 1.3 MB 00:03 2024-03-25T18:10:22.213 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-25T18:10:22.217 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-25T18:10:22.217 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-25T18:10:22.262 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-25T18:10:22.262 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-25T18:10:22.553 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-25T18:10:22.661 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:23.073 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:10:23.324 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-25T18:10:23.531 DEBUG:teuthology.parallel:result is None 2024-03-25T18:15:27.168 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 290 kB/s | 287 MB 16:53 2024-03-25T18:15:27.168 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:27.168 INFO:teuthology.orchestra.run.smithi186.stdout:Total 290 kB/s | 287 MB 16:53 2024-03-25T18:15:27.169 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:15:27.172 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:15:27.172 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:15:28.879 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:15:28.879 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:15:58.360 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:15:58.470 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.603 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:15:59.900 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:16:00.120 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:16:00.685 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:56:30 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:16:00.833 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:00.834 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:16:00.835 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:00.835 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 5.9 M 2024-03-25T18:16:00.835 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 28 M 2024-03-25T18:16:00.835 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:16:16.673 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 384 kB/s | 5.9 MB 00:15 2024-03-25T18:16:16.674 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:16.674 INFO:teuthology.orchestra.run.smithi186.stdout:Total 384 kB/s | 5.9 MB 00:15 2024-03-25T18:16:16.675 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:16:16.679 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:16:16.679 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:16:16.750 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:16:16.750 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:16:17.485 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:18.140 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:18.425 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:16:18.599 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:16:19.162 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:56:49 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:16:19.314 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:19.315 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:16:19.316 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:19.316 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 109 k 2024-03-25T18:16:19.316 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 440 k 2024-03-25T18:16:19.316 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:16:20.143 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 131 kB/s | 109 kB 00:00 2024-03-25T18:16:20.144 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:20.144 INFO:teuthology.orchestra.run.smithi186.stdout:Total 131 kB/s | 109 kB 00:00 2024-03-25T18:16:20.144 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:16:20.147 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:16:20.148 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:16:20.184 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:16:20.184 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:16:20.299 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:16:20.400 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:20.808 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:21.091 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:16:21.277 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:16:21.844 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:56:51 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:16:21.992 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:21.993 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 5.4 M 2024-03-25T18:16:21.994 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 25 M 2024-03-25T18:16:21.994 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:16:29.862 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 697 kB/s | 5.4 MB 00:07 2024-03-25T18:16:29.863 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:29.863 INFO:teuthology.orchestra.run.smithi186.stdout:Total 696 kB/s | 5.4 MB 00:07 2024-03-25T18:16:29.863 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:16:29.867 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:16:29.867 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:16:29.935 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:16:29.935 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:16:30.595 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:16:30.703 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:31.251 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:31.530 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:16:31.716 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2-debuginfo 2024-03-25T18:16:32.284 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:57:02 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:16:32.433 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:16:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:16:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:16:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 44 M 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 253 M 2024-03-25T18:16:32.435 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:18:36.076 INFO:teuthology.orchestra.run.smithi186.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 367 kB/s | 44 MB 02:03 2024-03-25T18:18:36.077 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:36.077 INFO:teuthology.orchestra.run.smithi186.stdout:Total 367 kB/s | 44 MB 02:03 2024-03-25T18:18:36.077 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:18:36.080 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:18:36.080 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:18:36.381 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:18:36.381 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:18:41.490 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:18:41.592 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:43.485 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:43.769 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:43.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:18:43.769 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:18:43.769 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:43.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:18:43.770 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:18:43.965 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:18:44.531 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 1:59:14 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:18:44.684 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:18:44.684 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:18:44.684 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:18:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 52 M 2024-03-25T18:18:44.686 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 308 M 2024-03-25T18:18:44.686 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:20:38.633 INFO:teuthology.orchestra.run.smithi186.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 466 kB/s | 52 MB 01:53 2024-03-25T18:20:38.634 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:38.634 INFO:teuthology.orchestra.run.smithi186.stdout:Total 466 kB/s | 52 MB 01:53 2024-03-25T18:20:38.634 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:20:38.636 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:20:38.637 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:20:38.980 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:20:38.981 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:20:44.986 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:20:45.213 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:47.283 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:47.574 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:47.575 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:47.575 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:20:47.575 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:47.575 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:47.575 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:20:47.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:20:48.322 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:18 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:20:48.470 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:48.471 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 615 k 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.1 M 2024-03-25T18:20:48.472 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:20:50.184 INFO:teuthology.orchestra.run.smithi186.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 359 kB/s | 615 kB 00:01 2024-03-25T18:20:50.185 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:50.185 INFO:teuthology.orchestra.run.smithi186.stdout:Total 359 kB/s | 615 kB 00:01 2024-03-25T18:20:50.186 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:20:50.190 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:20:50.190 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:20:50.231 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:20:50.232 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:20:50.405 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:20:50.501 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:50.914 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:51.184 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:51.184 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:51.184 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:20:51.185 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:51.185 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:51.185 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:20:51.368 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:20:51.940 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:21 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:20:52.089 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 771 k 2024-03-25T18:20:52.090 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.7 M 2024-03-25T18:20:52.091 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:20:54.029 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 398 kB/s | 771 kB 00:01 2024-03-25T18:20:54.030 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:54.030 INFO:teuthology.orchestra.run.smithi186.stdout:Total 397 kB/s | 771 kB 00:01 2024-03-25T18:20:54.030 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:20:54.034 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:20:54.034 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:20:54.075 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:20:54.075 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:20:54.261 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:20:54.361 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:54.758 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:55.050 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:55.050 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:55.050 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:20:55.050 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:55.050 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:55.051 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:20:55.262 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:20:55.828 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:25 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:20:55.978 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:20:55.978 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:20:55.979 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:20:55.980 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.1 M 2024-03-25T18:20:55.980 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 4.3 M 2024-03-25T18:20:55.980 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:21:00.023 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 285 kB/s | 1.1 MB 00:04 2024-03-25T18:21:00.024 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:00.024 INFO:teuthology.orchestra.run.smithi186.stdout:Total 284 kB/s | 1.1 MB 00:04 2024-03-25T18:21:00.025 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:21:00.029 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:21:00.029 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:21:00.072 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:21:00.073 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:21:00.299 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:21:00.396 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:00.801 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:01.083 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:21:01.299 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:21:01.869 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:31 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:21:02.020 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:21:02.020 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:02.020 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:21:02.020 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 337 k 2024-03-25T18:21:02.022 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.0 M 2024-03-25T18:21:02.022 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:21:03.085 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 317 kB/s | 337 kB 00:01 2024-03-25T18:21:03.085 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:03.086 INFO:teuthology.orchestra.run.smithi186.stdout:Total 317 kB/s | 337 kB 00:01 2024-03-25T18:21:03.086 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:21:03.089 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:21:03.090 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:21:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:21:03.129 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:21:03.265 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:21:03.376 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:03.764 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:04.057 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:21:04.262 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:21:04.832 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:34 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:21:04.982 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:21:04.983 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:04.984 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 509 k 2024-03-25T18:21:04.984 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.5 M 2024-03-25T18:21:04.984 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:21:07.138 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 236 kB/s | 509 kB 00:02 2024-03-25T18:21:07.138 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:07.139 INFO:teuthology.orchestra.run.smithi186.stdout:Total 236 kB/s | 509 kB 00:02 2024-03-25T18:21:07.139 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:21:07.143 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:21:07.143 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:21:07.183 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:21:07.183 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:21:07.346 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:21:07.453 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:07.834 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:08.108 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:21:08.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:21:08.878 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:01:38 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:21:09.030 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 52 M 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 298 M 2024-03-25T18:21:09.031 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:23:06.570 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 449 kB/s | 52 MB 01:57 2024-03-25T18:23:06.570 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:06.570 INFO:teuthology.orchestra.run.smithi186.stdout:Total 449 kB/s | 52 MB 01:57 2024-03-25T18:23:06.570 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:23:06.573 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:23:06.573 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:23:06.905 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:23:06.905 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:23:12.792 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:23:13.041 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:15.067 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:15.356 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:23:15.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:23:16.126 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 2:03:46 ago on Mon 25 Mar 2024 04:19:30 PM UTC. 2024-03-25T18:23:16.274 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-25T18:23:16.275 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-25T18:23:16.276 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:16.276 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.3 M 2024-03-25T18:23:16.276 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 5.5 M 2024-03-25T18:23:16.276 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-25T18:23:20.703 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 290 kB/s | 1.3 MB 00:04 2024-03-25T18:23:20.704 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:20.704 INFO:teuthology.orchestra.run.smithi186.stdout:Total 289 kB/s | 1.3 MB 00:04 2024-03-25T18:23:20.704 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-25T18:23:20.708 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-25T18:23:20.708 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-25T18:23:20.753 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-25T18:23:20.753 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-25T18:23:21.014 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-25T18:23:21.119 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:21.528 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:21.805 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-25T18:23:22.007 DEBUG:teuthology.parallel:result is None 2024-03-25T18:23:22.008 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:23:22.227 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:23:22.252 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:22.252 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:22.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:23:22.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:23:22.469 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:23:22.490 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:22.490 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:22.491 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:23:22.492 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:23:22.492 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:22.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:23:22.523 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:22.523 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:23:22.559 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:23:22.559 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:22.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:23:22.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:23:22.662 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:22.662 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:23:22.691 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:23:22.760 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:23:22.760 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:22.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:23:22.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:23:22.857 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:22.857 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:23:22.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:23:22.954 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:23:22.954 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:22.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:23:22.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:23:23.050 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:23.050 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:23:23.079 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:23:23.147 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:23:23.225 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:23:23.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-03-25T18:23:23.228 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-03-25T18:23:23.256 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:23:23.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:23:23.296 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:23:23.323 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:23:23.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:23:23.363 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:23:23.425 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:23:23.425 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:23:23.426 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621252', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK6Y5chr1P3W+bPK6Xaeaip+/nsTZQ5qzyhjymkTeb+IGibowaAwcZ9ftmZrfeqMR0LnOXCZWuKsj9ExgiCMKQ=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfSSRsp6KvzxWvC/Wea72U/E4M7bNUtEnuPpOpE7RjfVCjbi2dR1vm7niezXKK3+lqWOkBM70RbsK8dCV/pf4I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:23:23.426 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:23:23.430 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:23:23.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:23:23.485 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:23:23.512 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:23.513 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:23:23.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:23:23.565 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-25 16:23:20.897224016 +0000 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-25 16:18:35.127891582 +0000 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-25 16:18:35.127891582 +0000 2024-03-25T18:23:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-25 16:18:35.127891582 +0000 2024-03-25T18:23:23.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:23:23.685 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-25T18:23:23.685 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-25T18:23:23.685 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000258214 s, 2.0 MB/s 2024-03-25T18:23:23.686 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:23:23.742 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-25 16:23:20.898224009 +0000 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-25 16:18:35.483888876 +0000 2024-03-25T18:23:23.797 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-25 16:18:35.483888876 +0000 2024-03-25T18:23:23.798 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-25 16:18:35.483888876 +0000 2024-03-25T18:23:23.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:23:23.863 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-25T18:23:23.863 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-25T18:23:23.863 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000192222 s, 2.7 MB/s 2024-03-25T18:23:23.864 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:23:23.920 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-25 16:23:20.898224009 +0000 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-25 16:18:35.819886322 +0000 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-25 16:18:35.819886322 +0000 2024-03-25T18:23:23.975 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-25 16:18:35.819886322 +0000 2024-03-25T18:23:23.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:23:24.041 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000168019 s, 3.0 MB/s 2024-03-25T18:23:24.043 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:23:24.098 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-25 16:23:20.898224009 +0000 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-25 16:18:36.170883654 +0000 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-25 16:18:36.170883654 +0000 2024-03-25T18:23:24.153 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-25 16:18:36.170883654 +0000 2024-03-25T18:23:24.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:23:24.218 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-25T18:23:24.218 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-25T18:23:24.218 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000198464 s, 2.6 MB/s 2024-03-25T18:23:24.219 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:23:24.275 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:24.275 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:24.275 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:23:24.291 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:23:24.292 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-25 16:24:03.599008645 +0000 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-25 16:18:34.901757000 +0000 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-25 16:18:34.901757000 +0000 2024-03-25T18:23:24.348 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-25 16:18:34.901757000 +0000 2024-03-25T18:23:24.349 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:23:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-25T18:23:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-25T18:23:24.417 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00022325 s, 2.3 MB/s 2024-03-25T18:23:24.418 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:23:24.476 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-03-25T18:23:24.532 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-25 16:24:03.599008645 +0000 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-25 16:18:35.235750932 +0000 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-25 16:18:35.235750932 +0000 2024-03-25T18:23:24.533 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-25 16:18:35.235750932 +0000 2024-03-25T18:23:24.533 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:23:24.600 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-25T18:23:24.600 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-25T18:23:24.600 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000307289 s, 1.7 MB/s 2024-03-25T18:23:24.601 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:23:24.658 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-03-25T18:23:24.714 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:23:24.714 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-25 16:24:03.600008627 +0000 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-25 16:18:35.578744701 +0000 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-25 16:18:35.578744701 +0000 2024-03-25T18:23:24.715 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-25 16:18:35.578744701 +0000 2024-03-25T18:23:24.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:23:24.781 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-25T18:23:24.781 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-25T18:23:24.781 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000192349 s, 2.7 MB/s 2024-03-25T18:23:24.782 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:23:24.839 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-03-25T18:23:24.894 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-25 16:24:03.600008627 +0000 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-25 16:18:35.925738397 +0000 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-25 16:18:35.925738397 +0000 2024-03-25T18:23:24.895 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-25 16:18:35.925738397 +0000 2024-03-25T18:23:24.895 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:23:24.961 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-25T18:23:24.961 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-25T18:23:24.961 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000187044 s, 2.7 MB/s 2024-03-25T18:23:24.962 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:23:25.017 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:25.017 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:23:25.017 INFO:tasks.ceph:Generating config... 2024-03-25T18:23:25.019 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:23:25.019 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:23:25.019 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:23:25.019 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:23:25.019 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:23:25.019 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:23:25.019 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:23:25.019 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:23:25.020 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:23:25.020 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:23:25.021 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:23:25.022 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:23:25.022 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:23:25.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:23:25.091 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:23:25.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:23:25.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:25.168 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.176:6789]'), ('mon.c', '[v1:172.21.15.176:6790]'), ('mon.b', '[v1:172.21.15.186:6789]')] 2024-03-25T18:23:25.168 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.176:6789],[v1:172.21.15.176:6790],[v1:172.21.15.186:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:23:25.169 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:25.169 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:23:25.223 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.176:6789]' --addv c '[v1:172.21.15.176:6790]' --addv b '[v1:172.21.15.186:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:25.308 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:25.308 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: generated fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:setting min_mon_release = quincy 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:epoch 0 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:min_mon_release 17 (quincy) 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:election_strategy: 3 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:23:25.309 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:23:25.310 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:23:25.324 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ffa5ed0f-6227-4dd7-b2d6-1675fe357662... 2024-03-25T18:23:25.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:23:25.367 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.415 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.416 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.417 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "[v1:172.21.15.176:6789],[v1:172.21.15.176:6790],[v1:172.21.15.186:6789]" 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: mon election default strategy = 3 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: ms bind msgr2 = False 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: ms type = async 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: osd heartbeat grace = 80 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-03-25T18:23:25.418 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-03-25T18:23:25.419 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = stupid 2024-03-25T18:23:25.420 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore zero block detection = True 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd class default list = * 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd class load list = * 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd client watch timeout = 120 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify cached snaps = True 2024-03-25T18:23:25.421 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify missing on start = True 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd fast shutdown = False 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd max markdown count = 1000 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock override recovery settings = True 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue = debug_random 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue cut off = debug_random 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: osd shutdown pgref assert = True 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-25T18:23:25.422 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:25.423 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon min osdmap epochs = 50 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd crush smoke test = False 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune interval = 2 2024-03-25T18:23:25.424 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune min = 15 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub interval = 300 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: paxos service trim min = 10 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: debug objecter = 20 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout: debug rados = 20 2024-03-25T18:23:25.425 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-03-25T18:23:25.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-03-25T18:23:25.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "[v1:172.21.15.176:6789],[v1:172.21.15.176:6790],[v1:172.21.15.186:6789]" 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon election default strategy = 3 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms bind msgr2 = False 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 5000 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms type = async 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd heartbeat grace = 80 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-03-25T18:23:25.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-03-25T18:23:25.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = stupid 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-03-25T18:23:25.431 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore zero block detection = True 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd class default list = * 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd class load list = * 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd client watch timeout = 120 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify cached snaps = True 2024-03-25T18:23:25.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify missing on start = True 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd fast shutdown = False 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock override recovery settings = True 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue = debug_random 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue cut off = debug_random 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd shutdown pgref assert = True 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.434 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:23:25.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon min osdmap epochs = 50 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd crush smoke test = False 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune interval = 2 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune min = 15 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon scrub interval = 300 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on pool no app = False 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: paxos service trim min = 10 2024-03-25T18:23:25.437 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: debug objecter = 20 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout: debug rados = 20 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-03-25T18:23:25.438 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-03-25T18:23:25.440 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:23:25.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:23:25.494 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:23:25.495 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:25.495 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:23:25.509 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:25.509 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:23:25.564 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:23:25.564 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:25.564 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:23:25.564 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:25.644 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:25.644 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:25.699 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:23:25.700 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:25.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:23:25.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:25.739 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:25.739 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:25.793 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:23:25.793 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:23:25.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:23:25.847 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:23:25.850 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:23:25.934 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:23:25.936 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:23:25.936 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:23:25.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:23:25.978 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:23:25.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:23:26.036 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:23:26.051 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:23:26.051 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:23:26.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:26.078 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:26.078 INFO:tasks.ceph:role: osd.0 2024-03-25T18:23:26.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:23:26.078 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:26.153 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:26.154 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:26.666 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-25T18:23:26.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-25T18:23:26.669 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:26.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:26.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:26.921 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:26.921 INFO:tasks.ceph:role: osd.1 2024-03-25T18:23:26.921 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:23:26.921 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:23:26.995 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:26.996 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:27.438 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-25T18:23:27.440 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-25T18:23:27.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:27.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:27.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:27.633 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:27.633 INFO:tasks.ceph:role: osd.2 2024-03-25T18:23:27.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:23:27.634 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:23:27.708 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:27.708 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:27.708 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:27.708 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:27.709 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:28.218 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-25T18:23:28.220 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-25T18:23:28.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:28.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:28.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:28.470 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:28.470 INFO:tasks.ceph:role: osd.3 2024-03-25T18:23:28.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:23:28.470 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:23:28.544 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:28.545 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:29.026 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-25T18:23:29.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-25T18:23:29.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:29.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:29.199 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.293+0000 7fe379d26640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.293+0000 7fe379d26640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.293+0000 7fe379d26640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.293+0000 7fe379d26640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.293+0000 7fe379d26640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:29.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:29.294+0000 7fe379d26640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:23:32.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:32.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:32.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.264+0000 7fe653e13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:23:32.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.265+0000 7fe653e13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:23:32.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.265+0000 7fe653e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:23:32.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.265+0000 7fe653e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:23:32.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.265+0000 7fe653e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:23:32.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:32.265+0000 7fe653e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:23:35.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:35.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:35.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.228+0000 7f8568015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:23:35.229 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.229+0000 7f8568015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:23:35.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.229+0000 7f8568015640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:23:35.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.229+0000 7f8568015640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:23:35.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.229+0000 7f8568015640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:23:35.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:35.229+0000 7f8568015640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:23:38.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:38.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:38.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.194+0000 7f98bfece640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:23:38.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.194+0000 7f98bfece640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:23:38.196 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.194+0000 7f98bfece640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:23:38.196 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.194+0000 7f98bfece640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:23:38.196 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.194+0000 7f98bfece640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:23:38.196 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:38.195+0000 7f98bfece640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:23:41.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:41.069 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:23:41.100 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:41.100 INFO:tasks.ceph:role: osd.4 2024-03-25T18:23:41.100 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:23:41.100 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:23:41.179 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:41.179 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:41.179 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:41.179 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:41.180 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:41.696 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-25T18:23:41.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-25T18:23:41.699 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:23:41.843 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:23:41.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:23:41.951 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:41.951 INFO:tasks.ceph:role: osd.5 2024-03-25T18:23:41.952 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:23:41.952 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:42.027 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:42.468 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-25T18:23:42.470 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-25T18:23:42.470 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:23:42.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:23:42.598 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:23:42.664 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:42.665 INFO:tasks.ceph:role: osd.6 2024-03-25T18:23:42.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:23:42.665 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:42.739 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:43.244 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-25T18:23:43.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-25T18:23:43.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:23:43.393 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:23:43.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:23:43.491 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:43.491 INFO:tasks.ceph:role: osd.7 2024-03-25T18:23:43.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:23:43.492 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:23:43.565 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:43.566 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:44.047 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-03-25T18:23:44.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-03-25T18:23:44.049 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:23:44.199 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:23:44.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:44.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.331+0000 7f02d1a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:23:44.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.331+0000 7f02d1a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:23:44.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.331+0000 7f02d1a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:23:44.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.331+0000 7f02d1a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:23:44.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.331+0000 7f02d1a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:23:44.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:44.333+0000 7f02d1a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:23:47.188 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:23:47.219 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.311+0000 7f46761bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.312+0000 7f46761bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.312+0000 7f46761bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.312+0000 7f46761bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.312+0000 7f46761bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:23:47.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:47.312+0000 7f46761bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:23:50.151 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:23:50.182 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.273+0000 7fc080015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.274+0000 7fc080015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.274+0000 7fc080015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.274+0000 7fc080015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.274+0000 7fc080015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:23:50.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:50.274+0000 7fc080015640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:23:53.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:23:53.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:53.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.241+0000 7fdf4101e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:23:53.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.241+0000 7fdf4101e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:23:53.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.241+0000 7fdf4101e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:23:53.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.241+0000 7fdf4101e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:23:53.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.241+0000 7fdf4101e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:23:53.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:23:53.242+0000 7fdf4101e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:23:56.083 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:23:56.114 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:23:56.114 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.114 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:23:56.145 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.145 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:23:56.217 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.217 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:23:56.288 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:23:56.358 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:23:56.428 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:23:56.460 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:23:56.530 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:23:56.600 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.600 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:23:56.670 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.670 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:23:56.738 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:23:56.738 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:23:56.756 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:23:56.756 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:23:56.794 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:23:56.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:23:56.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:23:56.836 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAdwQFmB6ONMhAAkQGYtr3ATyDYCmImUDSwlQ== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAhwQFmSMCHERAAq2ShraR7OHuZ5Tt9ZQqJ8g== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAkwQFmCZbVDxAA7vJQrDdU4memkyMfJr8q4Q== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAnwQFmCqOyDRAA5BWW3edN8c5H5aC6O86OKA== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAqwQFmGSykCxAAkR05R8UXDQHL0ATXmjNWlQ== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAdwQFmtRuxNxAA/LAhCu4dV+FaAT4QU6HeQw== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAwwQFmNaHRExAAzMhRu82AT4zj0tP3AftzzA== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAzwQFmlC2lEhAAgypmhVrLGk5Uu+2cE/l1pg== 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-03-25T18:23:56.837 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA2wQFm9B9eEBAAMBEIF7vfhh0/hO5yvicPdA== 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA5wQFmJzB1DhAAqUjifDXIRv36mOg7eLE1VQ== 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAdwQFm8DVcOhAAZ5OASJv6bjSR4ma6mNWMSQ== 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-25T18:23:56.838 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAewQFmpF8rAhAArtXCl7q9lYfS1tq5zP7Rlw== 2024-03-25T18:23:56.859 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2024-03-25T18:23:56.859 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAdwQFmB6ONMhAAkQGYtr3ATyDYCmImUDSwlQ== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAhwQFmSMCHERAAq2ShraR7OHuZ5Tt9ZQqJ8g== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkwQFmCZbVDxAA7vJQrDdU4memkyMfJr8q4Q== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAnwQFmCqOyDRAA5BWW3edN8c5H5aC6O86OKA== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAqwQFmGSykCxAAkR05R8UXDQHL0ATXmjNWlQ== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAdwQFmtRuxNxAA/LAhCu4dV+FaAT4QU6HeQw== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAwwQFmNaHRExAAzMhRu82AT4zj0tP3AftzzA== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAzwQFmlC2lEhAAgypmhVrLGk5Uu+2cE/l1pg== 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2024-03-25T18:23:56.860 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA2wQFm9B9eEBAAMBEIF7vfhh0/hO5yvicPdA== 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA5wQFmJzB1DhAAqUjifDXIRv36mOg7eLE1VQ== 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAdwQFm8DVcOhAAZ5OASJv6bjSR4ma6mNWMSQ== 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-03-25T18:23:56.861 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAewQFmpF8rAhAArtXCl7q9lYfS1tq5zP7Rlw== 2024-03-25T18:23:56.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:23:56.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:23:56.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:56.956 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.010 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.064 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:23:57.172 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:23:57.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.225 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.333 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.387 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:23:57.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:23:57.440 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:23:57.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:23:57.501 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:23:57.556 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:23:57.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:23:57.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:23:57.678 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:23:57.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:23:57.772 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:23:57.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:23:57.895 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:23:57.922 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:23:58.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:23:58.045 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:58.048 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:58.100 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:23:58.100 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:23:58.100 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:23:58.101 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:23:58.103 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:23:58.103 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:23:58.103 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:23:58.103 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:23:58.105 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:23:58.105 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:23:58.105 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:23:58.105 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:23:58.143 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:23:58.143 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:23:58.143 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:23:58.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:23:58.145 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:23:58.146 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:23:58.146 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:23:58.147 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:23:58.148 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:23:58.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.248+0000 7f5280ed9640 10 monclient: get_monmap_and_config 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.248+0000 7f5280ed9640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.248+0000 7f5280ed9640 10 monclient: monmap: 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:58.249405+0000 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:58.249405+0000 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:23:58.249 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:23:58.250 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 1 Processor -- start 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 1 -- start start 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient: init 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f527c0b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f527c142a20 addr v1:172.21.15.186:6789/0 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0a9d60 con 0x7f527c0b4700 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c13ff10 con 0x7f527c142a20 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0aa1f0 con 0x7f527c0a96b0 2024-03-25T18:23:58.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f5280ed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f527c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:23:58.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.250+0000 7f5280ed9640 10 monclient(hunting): authenticate will time out at 8116.561841s 2024-03-25T18:23:58.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.249+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f527c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:23:58.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.250+0000 7f5279d9b640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f527c142a20 legacy=0x7f527c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:23:58.451 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.451+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f527c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:23:58.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.451+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f527c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:23:58.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.451+0000 7f5279d9b640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f527c142a20 legacy=0x7f527c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:23:58.852 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.852+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f527c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:23:58.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.852+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f527c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:23:58.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:58.852+0000 7f5279d9b640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f527c142a20 legacy=0x7f527c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:23:59.653 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:59.653+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f527c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:23:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:59.653+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f527c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:23:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:23:59.653+0000 7f5279d9b640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f527c142a20 legacy=0x7f527c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.249+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.249+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.249+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.249+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.249+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f527c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f527c142a20 legacy=0x7f527c13f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f527c142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c003680 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f527c0b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b4ae0 con 0x7f527c0b4700 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0a9d60 con 0x7f525c003680 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c13ff10 con 0x7f527c0a96b0 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f527a59c640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.250+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:01.451 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.451+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:01.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.451+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:01.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.451+0000 7f527a59c640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:01.852 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.852+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:01.852 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.852+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:01.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:01.852+0000 7f527a59c640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:02.654 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:02.653+0000 7f527ad9d640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:02.654 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:02.653+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:02.654 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:02.653+0000 7f527a59c640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:04.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:04.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:04.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:04.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:04.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f527c0b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c003680 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0aa1f0 con 0x7f527c0b4700 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b4ae0 con 0x7f525c003680 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0a9d60 con 0x7f527c0a96b0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c0103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:04.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.250+0000 7f527ad9d640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:04.451 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.451+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:04.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.451+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c0103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:04.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.451+0000 7f527ad9d640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:04.852 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.852+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:04.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.852+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c0103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:04.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:04.852+0000 7f527ad9d640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:05.157 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:00:06.891 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.348 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:00:07.105 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.354 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:00:07.110 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.465 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:00:07.199 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.653 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:05.653+0000 7f5279d9b640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6789/0 2024-03-25T18:24:05.653 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:05.653+0000 7f527a59c640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c0103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.176:6790/0 2024-03-25T18:24:05.653 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:05.653+0000 7f527ad9d640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-03-25T18:24:05.657 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:00:07.414 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:05.664 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:00:07.420 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:07.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:07.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:07.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c0103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c00a310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c003680 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f527c0b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c13ff10 con 0x7f527c0b4700 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0aa1f0 con 0x7f525c003680 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b4ae0 con 0x7f527c0a96b0 2024-03-25T18:24:07.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.250+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:07.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.329+0000 7f527ad9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 0x7f525c0103a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:53674/0 (socket says 172.21.15.176:53674) 2024-03-25T18:24:07.329 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:07.329+0000 7f527ad9d640 1 -- 172.21.15.176:0/1306005807 learned_addr learned my addr 172.21.15.176:0/1306005807 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:10.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:10.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c003680 legacy=0x7f525c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f527c0b4700 legacy=0x7f525c0103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c0124f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0a9d60 con 0x7f525c0124f0 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00d240 con 0x7f525c00cc70 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00d490 con 0x7f527c0a96b0 2024-03-25T18:24:10.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:10.251+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f527c0a96b0 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c00a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c0124f0 legacy=0x7f525c0148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f525c010680 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c00de70 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c010470 con 0x7f525c00de70 2024-03-25T18:24:13.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00a370 con 0x7f525c00cc70 2024-03-25T18:24:13.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00a5a0 con 0x7f525c010680 2024-03-25T18:24:13.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:13.251+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f525c010680 legacy=0x7f525c00d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c00de70 legacy=0x7f525c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c0092c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f525c009ed0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00a7d0 con 0x7f525c0092c0 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00d6c0 con 0x7f525c00cc70 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00d8f0 con 0x7f525c009ed0 2024-03-25T18:24:16.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:16.251+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:19.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:19.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:19.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:19.252 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f525c009ed0 legacy=0x7f525c0158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c0092c0 legacy=0x7f525c009700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c00bcc0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f525c009ed0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00db20 con 0x7f525c00bcc0 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00c8f0 con 0x7f525c00cc70 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c0096a0 con 0x7f525c009ed0 2024-03-25T18:24:19.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:19.252+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.251+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.251+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.251+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.251+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.251+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f525c009ed0 legacy=0x7f525c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c00bcc0 legacy=0x7f525c00c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f525c006200 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c006d50 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:22.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:22.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c0098d0 con 0x7f525c006d50 2024-03-25T18:24:22.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c009b20 con 0x7f525c00cc70 2024-03-25T18:24:22.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00c0a0 con 0x7f525c006200 2024-03-25T18:24:22.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.252+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.255+0000 7f5279d9b640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 legacy=0x7f525c0158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:24:22.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.255+0000 7f5279d9b640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 legacy=0x7f525c0158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:24:22.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.255+0000 7f5279d9b640 1 --1- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 0x7f525c0158f0 :-1 s=OPENED pgs=26 cs=1 l=1).handle_message read tag failed 2024-03-25T18:24:22.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.255+0000 7f5279d9b640 1 --1- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 0x7f525c0158f0 :-1 s=OPENED pgs=26 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:24:22.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:22.255+0000 7f527959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6789/0 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): tick 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f525c006200 legacy=0x7f525c006640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 legacy=0x7f525c0158f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f525c00c3c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f525c00cc70 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f525c006d50 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:25.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c00c230 con 0x7f525c006d50 2024-03-25T18:24:25.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c0158f0 con 0x7f525c00cc70 2024-03-25T18:24:25.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c015ad0 con 0x7f525c00c3c0 2024-03-25T18:24:25.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:25.252+0000 7f526bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460833234 0 0) 0x7f525c00c230 con 0x7f525c006d50 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:24:26.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.295+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5264003520 con 0x7f525c006d50 2024-03-25T18:24:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113509511 0 0) 0x7f525c0158f0 con 0x7f525c00cc70 2024-03-25T18:24:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:24:26.305 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.303+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c00c230 con 0x7f525c00cc70 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668201457 0 0) 0x7f525c015ad0 con 0x7f525c00c3c0 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:24:26.332 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.331+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c0158f0 con 0x7f525c00c3c0 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.349+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108492294 0 0) 0x7f5264003520 con 0x7f525c006d50 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.349+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6790/0 conn(0x7f525c00c3c0 legacy=0x7f525c016000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.349+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.186:6789/0 conn(0x7f525c00cc70 legacy=0x7f525c010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 1 monclient: found mon.noname-a 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c015d00 con 0x7f525c006d50 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: _finish_auth 0 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f52740041d0 con 0x7f525c006d50 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f5280ed9640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f5280ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:26.351 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f527959a640 10 monclient: dump: 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:26.352 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.350+0000 7f5280ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:26.389 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52740043a0 con 0x7f525c006d50 2024-03-25T18:24:26.389 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:26.389 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 1 -- 172.21.15.176:0/1306005807 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f52740047b0 con 0x7f525c006d50 2024-03-25T18:24:26.389 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f527959a640 10 monclient: dump: 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:26.390 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:26.391 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 4 set_mon_vals no callback set 2024-03-25T18:24:26.393 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 10 monclient: shutdown 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 1 -- 172.21.15.176:0/1306005807 >> v1:172.21.15.176:6789/0 conn(0x7f525c006d50 legacy=0x7f525c006640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 1 -- 172.21.15.176:0/1306005807 shutdown_connections 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 1 -- 172.21.15.176:0/1306005807 >> 172.21.15.176:0/1306005807 conn(0x7f527c0668e0 msgr2=0x7f527c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 1 -- 172.21.15.176:0/1306005807 shutdown_connections 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.389+0000 7f5280ed9640 1 -- 172.21.15.176:0/1306005807 wait complete. 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.390+0000 7f5280ed9640 10 monclient: build_initial_monmap 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.390+0000 7f5280ed9640 10 monclient: monmap: 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:26.394 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 librados: starting msgr at 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 librados: starting objecter 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 Processor -- start 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 -- start start 2024-03-25T18:24:26.395 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 librados: setting wanted keys 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 librados: calling monclient init 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient: init 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f525c006d50 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f525c00cc70 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f527c0b1560 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b03e0 con 0x7f525c00cc70 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b06c0 con 0x7f525c006d50 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c1d7830 con 0x7f527c0b1560 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.391+0000 7f5280ed9640 10 monclient(hunting): authenticate will time out at 8144.703633s 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f527ad9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f527c0b1560 0x7f527c1e21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48672/0 (socket says 172.21.15.176:48672) 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f527ad9d640 1 -- 172.21.15.176:0/3528395590 learned_addr learned my addr 172.21.15.176:0/3528395590 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:26.396 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f526affd640 10 client.?.objecter ms_handle_connect 0x7f527c0b1560 2024-03-25T18:24:26.397 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f526affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:26.397 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f526affd640 10 client.?.objecter ms_handle_connect 0x7f525c006d50 2024-03-25T18:24:26.397 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.392+0000 7f526affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:26.438 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.437+0000 7f526affd640 10 client.?.objecter ms_handle_connect 0x7f525c00cc70 2024-03-25T18:24:26.438 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.437+0000 7f526affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:26.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637172610 0 0) 0x7f525c00c230 con 0x7f525c00cc70 2024-03-25T18:24:26.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:24:26.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.442+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c015ad0 con 0x7f525c00cc70 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357531010 0 0) 0x7f525c0158f0 con 0x7f525c006d50 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:24:26.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.456+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c00c230 con 0x7f525c006d50 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424184534 0 0) 0x7f525c015d00 con 0x7f527c0b1560 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.458+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c0158f0 con 0x7f527c0b1560 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1656628526 0 0) 0x7f525c015ad0 con 0x7f525c00cc70 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f525c015d00 con 0x7f525c00cc70 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5270004140 con 0x7f525c00cc70 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 10 monclient(hunting): dump: 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.468 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:26.469 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.467+0000 7f526affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 869716381 0 0) 0x7f525c015d00 con 0x7f525c00cc70 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 >> v1:172.21.15.176:6790/0 conn(0x7f527c0b1560 legacy=0x7f527c1e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 >> v1:172.21.15.186:6789/0 conn(0x7f525c006d50 legacy=0x7f527c1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 1 monclient: found mon.a 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:26.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c0ae060 con 0x7f525c00cc70 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 10 monclient: _finish_auth 0 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f526affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 10 monclient: _renew_subs 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c1d7830 con 0x7f525c00cc70 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 10 monclient: _renew_subs 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c1d7830 con 0x7f525c00cc70 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.524+0000 7f5280ed9640 1 librados: init done 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.525+0000 7f5280ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.525+0000 7f5280ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.525+0000 7f5280ed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.528 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.525+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c04f730 con 0x7f525c00cc70 2024-03-25T18:24:26.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5270004550 con 0x7f525c00cc70 2024-03-25T18:24:26.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:26.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f52700049a0 con 0x7f525c00cc70 2024-03-25T18:24:26.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: dump: 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5270004f50 con 0x7f525c00cc70 2024-03-25T18:24:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1545963522 0 0) 0x7f5270006060 con 0x7f525c00cc70 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 client.4107.objecter ms_dispatch 0x7f527c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f5278d99640 4 set_mon_vals no callback set 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5270006680 con 0x7f525c00cc70 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.736+0000 7f526affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:26.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.823+0000 7f5280ed9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:26.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.823+0000 7f5280ed9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:26.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.823+0000 7f5280ed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:26.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:26.823+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f527c04f730 con 0x7f525c00cc70 2024-03-25T18:24:27.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.119+0000 7f526affd640 1 -- 172.21.15.176:0/3528395590 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f52700052e0 con 0x7f525c00cc70 2024-03-25T18:24:27.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.119+0000 7f526affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:27.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.119+0000 7f526affd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:24:27.121 INFO:teuthology.orchestra.run.smithi176.stderr:adjusted tunables profile to default 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.120+0000 7f5280ed9640 10 librados: watch_flush enter 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.120+0000 7f5280ed9640 10 librados: watch_flush exit 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.120+0000 7f5280ed9640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.120+0000 7f5280ed9640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 10 monclient: shutdown 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 >> v1:172.21.15.176:6789/0 conn(0x7f525c00cc70 legacy=0x7f527c0b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 shutdown_connections 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 >> 172.21.15.176:0/3528395590 conn(0x7f527c0668e0 msgr2=0x7f527c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:27.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 shutdown_connections 2024-03-25T18:24:27.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 -- 172.21.15.176:0/3528395590 wait complete. 2024-03-25T18:24:27.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:27.121+0000 7f5280ed9640 1 librados: shutdown 2024-03-25T18:24:27.134 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:24:27.134 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:24:27.134 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:24:27.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:24:27.162 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:24:27.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:24:27.230 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:24:27.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:24:27.245 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:27.244+0000 7fc9c5a40040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:24:27.292 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:27.291+0000 7f9d251b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:24:27.298 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-25T18:24:27.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:24:27.312 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:27.312+0000 7fc9c5a40040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:24:27.354 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:27.354+0000 7f9d251b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:24:27.366 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:24:27.366 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:24:27.393 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:24:27.393 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:24:27.460 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:24:27.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:24:27.528 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-25T18:24:27.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:24:27.594 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 946c70d1-508e-40ef-bd1d-a2db16c013ae 0 2024-03-25T18:24:27.734 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.733+0000 7f81af4a8640 10 monclient: get_monmap_and_config 2024-03-25T18:24:27.734 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.733+0000 7f81af4a8640 10 monclient: build_initial_monmap 2024-03-25T18:24:27.734 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.733+0000 7f81af4a8640 10 monclient: monmap: 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:27.734720+0000 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:27.734720+0000 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:27.735 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.734+0000 7f81af4a8640 1 Processor -- start 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.734+0000 7f81af4a8640 1 -- start start 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.734+0000 7f81af4a8640 10 monclient: init 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f81a80b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f81a8142a20 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f81a80a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80a9d60 con 0x7f81a80b4700 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813ff10 con 0x7f81a8142a20 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80aa1f0 con 0x7f81a80a96b0 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81af4a8640 10 monclient(hunting): authenticate will time out at 8144.120702s 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81ada45640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f81a80a96b0 0x7f81a813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44188/0 (socket says 172.21.15.186:44188) 2024-03-25T18:24:27.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.735+0000 7f81ada45640 1 -- 172.21.15.186:0/677852385 learned_addr learned my addr 172.21.15.186:0/677852385 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493756173 0 0) 0x7f81a80aa1f0 con 0x7f81a80a96b0 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:24:27.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.738+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8194003520 con 0x7f81a80a96b0 2024-03-25T18:24:27.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903956471 0 0) 0x7f81a80a9d60 con 0x7f81a80b4700 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.739+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a80aa1f0 con 0x7f81a80b4700 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092839184 0 0) 0x7f81a813ff10 con 0x7f81a8142a20 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:24:27.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.740+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a80a9d60 con 0x7f81a8142a20 2024-03-25T18:24:27.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793614703 0 0) 0x7f8194003520 con 0x7f81a80a96b0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 >> v1:172.21.15.186:6789/0 conn(0x7f81a8142a20 legacy=0x7f81a813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 >> v1:172.21.15.176:6789/0 conn(0x7f81a80b4700 legacy=0x7f81a8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 monclient: found mon.noname-b 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a80b4ae0 con 0x7f81a80a96b0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f81a0001a10 con 0x7f81a80a96b0 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f81af4a8640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f81af4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f819ffff640 10 monclient: dump: 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.742+0000 7f81af4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0001b70 con 0x7f81a80a96b0 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 1 -- 172.21.15.186:0/677852385 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f81a0001f80 con 0x7f81a80a96b0 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f819ffff640 10 monclient: dump: 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.770 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.771 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.771 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.771 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:27.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 4 set_mon_vals no callback set 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 10 monclient: shutdown 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 1 -- 172.21.15.186:0/677852385 >> v1:172.21.15.176:6790/0 conn(0x7f81a80a96b0 legacy=0x7f81a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 1 -- 172.21.15.186:0/677852385 shutdown_connections 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.769+0000 7f81af4a8640 1 -- 172.21.15.186:0/677852385 >> 172.21.15.186:0/677852385 conn(0x7f81a80668e0 msgr2=0x7f81a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 1 -- 172.21.15.186:0/677852385 shutdown_connections 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 1 -- 172.21.15.186:0/677852385 wait complete. 2024-03-25T18:24:27.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 10 monclient: build_initial_monmap 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 10 monclient: monmap: 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 1 librados: starting msgr at 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.770+0000 7f81af4a8640 1 librados: starting objecter 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 Processor -- start 2024-03-25T18:24:27.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 -- start start 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 librados: setting wanted keys 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 librados: calling monclient init 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient: init 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f81a80a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f81a80b4700 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f81a8142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a81377b0 con 0x7f81a8142a20 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a81379e0 con 0x7f81a80b4700 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a812ccc0 con 0x7f81a80a96b0 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.771+0000 7f81af4a8640 10 monclient(hunting): authenticate will time out at 8144.156988s 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f81aca43640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f81a80b4700 0x7f81a812d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39242/0 (socket says 172.21.15.186:39242) 2024-03-25T18:24:27.775 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f81aca43640 1 -- 172.21.15.186:0/4178418042 learned_addr learned my addr 172.21.15.186:0/4178418042 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a80b4700 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a80a96b0 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a8142a20 2024-03-25T18:24:27.776 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.772+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169682036 0 0) 0x7f81a81379e0 con 0x7f81a80b4700 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a80b4700 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164007746 0 0) 0x7f81a812ccc0 con 0x7f81a80a96b0 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:24:27.777 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.774+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a81379e0 con 0x7f81a80a96b0 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130020439 0 0) 0x7f81a81377b0 con 0x7f81a8142a20 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:24:27.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.775+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a812ccc0 con 0x7f81a8142a20 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971240673 0 0) 0x7f818c004eb0 con 0x7f81a80b4700 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a81377b0 con 0x7f81a80b4700 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f81a4001a10 con 0x7f81a80b4700 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3339245959 0 0) 0x7f81a81379e0 con 0x7f81a80a96b0 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a80a96b0 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8198001970 con 0x7f81a80a96b0 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.779+0000 7f819dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.781 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2751091346 0 0) 0x7f81a812ccc0 con 0x7f81a8142a20 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a81379e0 con 0x7f81a8142a20 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f81a0001e80 con 0x7f81a8142a20 2024-03-25T18:24:27.782 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.780+0000 7f819dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.783 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2122330544 0 0) 0x7f81a81379e0 con 0x7f81a8142a20 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 >> v1:172.21.15.176:6790/0 conn(0x7f81a80a96b0 legacy=0x7f81a813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 >> v1:172.21.15.186:6789/0 conn(0x7f81a80b4700 legacy=0x7f81a812d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 1 monclient: found mon.a 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:27.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8136880 con 0x7f81a8142a20 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f819dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 10 monclient: _renew_subs 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a813cab0 con 0x7f81a8142a20 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 10 monclient: _renew_subs 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8135570 con 0x7f81a8142a20 2024-03-25T18:24:27.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.783+0000 7f81af4a8640 1 librados: init done 2024-03-25T18:24:27.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.784+0000 7f81af4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:27.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.784+0000 7f81af4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:27.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.784+0000 7f81af4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:27.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.784+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a80b77a0 con 0x7f81a8142a20 2024-03-25T18:24:27.882 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:27.882+0000 7fc9c5a40040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0001560 con 0x7f81a8142a20 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f81a0004e10 con 0x7f81a8142a20 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 10 monclient: dump: 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.885 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f81a0005300 con 0x7f81a8142a20 2024-03-25T18:24:27.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.884+0000 7f819f7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1031917460 0 0) 0x7f81a0006750 con 0x7f81a8142a20 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 10 client.4119.objecter ms_dispatch 0x7f81a8000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f81a0006a40 con 0x7f81a8142a20 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:27.887 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.885+0000 7f819dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:27.913 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:27.913+0000 7f9d251b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:24:27.939 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:27.938+0000 7fc9c5a40040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:24:27.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.958+0000 7f81af4a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "946c70d1-508e-40ef-bd1d-a2db16c013ae", "id": 0}] 2024-03-25T18:24:27.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.958+0000 7f81af4a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "946c70d1-508e-40ef-bd1d-a2db16c013ae", "id": 0}] 2024-03-25T18:24:27.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.958+0000 7f81af4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:27.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:27.958+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "946c70d1-508e-40ef-bd1d-a2db16c013ae", "id": 0} v 0) v1 -- 0x7f81a80b3670 con 0x7f81a8142a20 2024-03-25T18:24:27.967 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:27.966+0000 7f9d251b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:24:27.983 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:27.983+0000 7fc9c5a40040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:24:28.013 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.013+0000 7f9d251b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:24:28.039 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.038+0000 7fc9c5a40040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:24:28.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.044+0000 7f819dffb640 1 -- 172.21.15.186:0/4178418042 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "946c70d1-508e-40ef-bd1d-a2db16c013ae", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 676370141 0 732707515) 0x7f81a0005910 con 0x7f81a8142a20 2024-03-25T18:24:28.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.044+0000 7f819dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "946c70d1-508e-40ef-bd1d-a2db16c013ae", "id": 0}] 2024-03-25T18:24:28.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.044+0000 7f819dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:28.046 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.045+0000 7f81af4a8640 10 librados: watch_flush enter 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.045+0000 7f81af4a8640 10 librados: watch_flush exit 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.045+0000 7f81af4a8640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.045+0000 7f81af4a8640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 10 monclient: shutdown 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 >> v1:172.21.15.176:6789/0 conn(0x7f81a8142a20 legacy=0x7f81a812d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 shutdown_connections 2024-03-25T18:24:28.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 >> 172.21.15.186:0/4178418042 conn(0x7f81a80668e0 msgr2=0x7f81a80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 shutdown_connections 2024-03-25T18:24:28.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 -- 172.21.15.186:0/4178418042 wait complete. 2024-03-25T18:24:28.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.046+0000 7f81af4a8640 1 librados: shutdown 2024-03-25T18:24:28.060 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e 1 2024-03-25T18:24:28.075 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.074+0000 7f9d251b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:24:28.085 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.084+0000 7fc9c5a40040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:24:28.120 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.119+0000 7f9d251b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.159+0000 7f407e881640 10 monclient: get_monmap_and_config 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.159+0000 7f407e881640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.159+0000 7f407e881640 10 monclient: monmap: 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:28.160349+0000 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:28.160349+0000 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:28.160 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:28.161 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 1 Processor -- start 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 1 -- start start 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient: init 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient(hunting): picked mon.noname-a con 0x7f40780b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient(hunting): picked mon.noname-b con 0x7f4078142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient(hunting): picked mon.noname-c con 0x7f40780a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40780a9d60 con 0x7f40780b4700 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407813ff10 con 0x7f40780a96b0 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40780aa1f0 con 0x7f4078142a20 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.160+0000 7f407e881640 10 monclient(hunting): authenticate will time out at 8144.546143s 2024-03-25T18:24:28.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.161+0000 7f407ce1e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f40780a96b0 0x7f407813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39244/0 (socket says 172.21.15.186:39244) 2024-03-25T18:24:28.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.161+0000 7f407ce1e640 1 -- 172.21.15.186:0/29365150 learned_addr learned my addr 172.21.15.186:0/29365150 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55735533 0 0) 0x7f407813ff10 con 0x7f40780a96b0 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4060003520 con 0x7f40780a96b0 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921700284 0 0) 0x7f40780a9d60 con 0x7f40780b4700 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.163+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f407813ff10 con 0x7f40780b4700 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922743425 0 0) 0x7f40780aa1f0 con 0x7f4078142a20 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:24:28.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.164+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40780a9d60 con 0x7f4078142a20 2024-03-25T18:24:28.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492132538 0 0) 0x7f407813ff10 con 0x7f40780b4700 2024-03-25T18:24:28.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 >> v1:172.21.15.176:6790/0 conn(0x7f4078142a20 legacy=0x7f407813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 >> v1:172.21.15.186:6789/0 conn(0x7f40780a96b0 legacy=0x7f407813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 monclient: found mon.noname-a 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780b4ae0 con 0x7f40780b4700 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4068001670 con 0x7f40780b4700 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f4076ffd640 10 monclient: dump: 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.168 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f407e881640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:24:28.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.166+0000 7f407e881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40680017d0 con 0x7f40780b4700 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 1 -- 172.21.15.186:0/29365150 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4068001be0 con 0x7f40780b4700 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f407e881640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f407e881640 4 set_mon_vals no callback set 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.168+0000 7f4076ffd640 10 monclient: dump: 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.171 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 10 monclient: shutdown 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 1 -- 172.21.15.186:0/29365150 >> v1:172.21.15.176:6789/0 conn(0x7f40780b4700 legacy=0x7f4078142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 1 -- 172.21.15.186:0/29365150 shutdown_connections 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 1 -- 172.21.15.186:0/29365150 >> 172.21.15.186:0/29365150 conn(0x7f40780668e0 msgr2=0x7f4078066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 1 -- 172.21.15.186:0/29365150 shutdown_connections 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 1 -- 172.21.15.186:0/29365150 wait complete. 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.169+0000 7f407e881640 10 monclient: monmap: 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:28.172 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 librados: starting msgr at 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 librados: starting objecter 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 Processor -- start 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 -- start start 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 librados: setting wanted keys 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 librados: calling monclient init 2024-03-25T18:24:28.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient: init 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient(hunting): picked mon.noname-a con 0x7f40780a96b0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient(hunting): picked mon.noname-b con 0x7f40780b4700 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient(hunting): picked mon.noname-c con 0x7f4078142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4078139410 con 0x7f40780a96b0 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4078131720 con 0x7f40780b4700 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40781318b0 con 0x7f4078142a20 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.170+0000 7f407e881640 10 monclient(hunting): authenticate will time out at 8144.556293s 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f40777fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f40780b4700 0x7f4078131010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39254/0 (socket says 172.21.15.186:39254) 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f40777fe640 1 -- 172.21.15.186:0/200625545 learned_addr learned my addr 172.21.15.186:0/200625545 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter ms_handle_connect 0x7f40780b4700 2024-03-25T18:24:28.176 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter ms_handle_connect 0x7f40780a96b0 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter ms_handle_connect 0x7f4078142a20 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.171+0000 7f4074ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629878544 0 0) 0x7f4078139410 con 0x7f40780a96b0 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4064004eb0 con 0x7f40780a96b0 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445308832 0 0) 0x7f4078131720 con 0x7f40780b4700 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:24:28.177 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4078139410 con 0x7f40780b4700 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103601083 0 0) 0x7f40781318b0 con 0x7f4078142a20 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:24:28.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.173+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4078131720 con 0x7f4078142a20 2024-03-25T18:24:28.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3956956497 0 0) 0x7f4064004eb0 con 0x7f40780a96b0 2024-03-25T18:24:28.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40781318b0 con 0x7f40780a96b0 2024-03-25T18:24:28.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4068001ba0 con 0x7f40780a96b0 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615935375 0 0) 0x7f4078139410 con 0x7f40780b4700 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4064004eb0 con 0x7f40780b4700 2024-03-25T18:24:28.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f406c001950 con 0x7f40780b4700 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484515648 0 0) 0x7f4078131720 con 0x7f4078142a20 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4078139410 con 0x7f4078142a20 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4070001cd0 con 0x7f4078142a20 2024-03-25T18:24:28.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.178+0000 7f4074ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.182 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708062885 0 0) 0x7f40781318b0 con 0x7f40780a96b0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 >> v1:172.21.15.176:6790/0 conn(0x7f4078142a20 legacy=0x7f40781d75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 >> v1:172.21.15.186:6789/0 conn(0x7f40780b4700 legacy=0x7f4078131010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 1 monclient: found mon.a 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4078138760 con 0x7f40780a96b0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f4074ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 10 monclient: _renew_subs 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4078138a10 con 0x7f40780a96b0 2024-03-25T18:24:28.184 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 10 monclient: _renew_subs 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4078135900 con 0x7f40780a96b0 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.181+0000 7f407e881640 1 librados: init done 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.182+0000 7f407e881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.182+0000 7f407e881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.182+0000 7f407e881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.182+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40781396f0 con 0x7f40780a96b0 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40680038a0 con 0x7f40780a96b0 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4068003a10 con 0x7f40780a96b0 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 10 monclient: dump: 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.186 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4068003f00 con 0x7f40780a96b0 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f40767fc640 4 set_mon_vals no callback set 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1945528325 0 0) 0x7f40680053c0 con 0x7f40780a96b0 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 10 client.4125.objecter ms_dispatch 0x7f4078000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:24:28.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.185+0000 7f4074ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:24:28.208 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.207+0000 7fc9c5a40040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:24:28.241 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.241+0000 7f9d251b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:24:28.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.278+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4068005740 con 0x7f40780a96b0 2024-03-25T18:24:28.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.278+0000 7f4074ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.278+0000 7f4074ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:28.279 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.279+0000 7fc9c5a40040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:28.313 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.311+0000 7f9d251b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:28.327 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.326+0000 7fc9c5a40040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:24:28.351 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.351+0000 7f407e881640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e", "id": 1}] 2024-03-25T18:24:28.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.351+0000 7f407e881640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e", "id": 1}] 2024-03-25T18:24:28.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.351+0000 7f407e881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.351+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e", "id": 1} v 0) v1 -- 0x7f40780ab3a0 con 0x7f40780a96b0 2024-03-25T18:24:28.357 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.356+0000 7f9d251b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:24:28.380 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.380+0000 7fc9c5a40040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:24:28.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.386+0000 7f4074ff9640 1 -- 172.21.15.186:0/200625545 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 43251027 0 940483276) 0x7f40680058e0 con 0x7f40780a96b0 2024-03-25T18:24:28.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.386+0000 7f4074ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e", "id": 1}] 2024-03-25T18:24:28.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.386+0000 7f4074ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:28.387 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.387+0000 7f407e881640 10 librados: watch_flush enter 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 10 librados: watch_flush exit 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 10 monclient: shutdown 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 >> v1:172.21.15.176:6789/0 conn(0x7f40780a96b0 legacy=0x7f4078138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 shutdown_connections 2024-03-25T18:24:28.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 >> 172.21.15.186:0/200625545 conn(0x7f40780668e0 msgr2=0x7f407813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 shutdown_connections 2024-03-25T18:24:28.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 -- 172.21.15.186:0/200625545 wait complete. 2024-03-25T18:24:28.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.388+0000 7f407e881640 1 librados: shutdown 2024-03-25T18:24:28.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 62facedd-b505-4f2b-a30a-86f1e7366d4f 2 2024-03-25T18:24:28.414 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.413+0000 7f9d251b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:24:28.431 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.430+0000 7fc9c5a40040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:24:28.461 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.461+0000 7f9d251b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:24:28.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.529+0000 7f1a308b3640 10 monclient: get_monmap_and_config 2024-03-25T18:24:28.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.529+0000 7f1a308b3640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.529+0000 7f1a308b3640 10 monclient: monmap: 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:28.530878+0000 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:28.530878+0000 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:28.531 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.532 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.530+0000 7f1a308b3640 1 Processor -- start 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.530+0000 7f1a308b3640 1 -- start start 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.530+0000 7f1a308b3640 10 monclient: init 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c0b4700 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c0a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0a9d60 con 0x7f1a2c142a20 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c13ff10 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0aa1f0 con 0x7f1a2c0b4700 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a308b3640 10 monclient(hunting): authenticate will time out at 8144.916639s 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a2ad9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1a2c0a96b0 0x7f1a2c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39258/0 (socket says 172.21.15.186:39258) 2024-03-25T18:24:28.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.531+0000 7f1a2ad9d640 1 -- 172.21.15.186:0/3079422563 learned_addr learned my addr 172.21.15.186:0/3079422563 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154020666 0 0) 0x7f1a2c13ff10 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a18003520 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752289564 0 0) 0x7f1a2c0a9d60 con 0x7f1a2c142a20 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.535 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c13ff10 con 0x7f1a2c142a20 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048680618 0 0) 0x7f1a2c0aa1f0 con 0x7f1a2c0b4700 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:24:28.536 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.534+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c0a9d60 con 0x7f1a2c0b4700 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304464280 0 0) 0x7f1a18003520 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 >> v1:172.21.15.176:6790/0 conn(0x7f1a2c0b4700 legacy=0x7f1a2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 >> v1:172.21.15.176:6789/0 conn(0x7f1a2c142a20 legacy=0x7f1a2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.538 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0b4ae0 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a24001a10 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a29d9b640 10 monclient: dump: 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.539 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.540 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.540 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a308b3640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:24:28.540 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.537+0000 7f1a308b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a24001b70 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 1 -- 172.21.15.186:0/3079422563 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a24001f80 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a29d9b640 10 monclient: dump: 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.571 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:28.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 4 set_mon_vals no callback set 2024-03-25T18:24:28.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 10 monclient: shutdown 2024-03-25T18:24:28.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.574 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 1 -- 172.21.15.186:0/3079422563 >> v1:172.21.15.186:6789/0 conn(0x7f1a2c0a96b0 legacy=0x7f1a2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 1 -- 172.21.15.186:0/3079422563 shutdown_connections 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 1 -- 172.21.15.186:0/3079422563 >> 172.21.15.186:0/3079422563 conn(0x7f1a2c0668e0 msgr2=0x7f1a2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 1 -- 172.21.15.186:0/3079422563 shutdown_connections 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.570+0000 7f1a308b3640 1 -- 172.21.15.186:0/3079422563 wait complete. 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.571+0000 7f1a308b3640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.571+0000 7f1a308b3640 10 monclient: monmap: 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.575 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 librados: starting msgr at 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 librados: starting objecter 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 Processor -- start 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 -- start start 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 librados: setting wanted keys 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 librados: calling monclient init 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient: init 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c0a96b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c0b4700 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c13b110 con 0x7f1a2c142a20 2024-03-25T18:24:28.576 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c1dbe60 con 0x7f1a2c0b4700 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0abc90 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.572+0000 7f1a308b3640 10 monclient(hunting): authenticate will time out at 8144.958210s 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.573+0000 7f1a2a59c640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f1a2c0a96b0 0x7f1a2c1db040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44230/0 (socket says 172.21.15.186:44230) 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.573+0000 7f1a2a59c640 1 -- 172.21.15.186:0/1338857370 learned_addr learned my addr 172.21.15.186:0/1338857370 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.573+0000 7f1a22ffd640 10 client.?.objecter ms_handle_connect 0x7f1a2c0a96b0 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.573+0000 7f1a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.574+0000 7f1a22ffd640 10 client.?.objecter ms_handle_connect 0x7f1a2c0b4700 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.574+0000 7f1a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.577 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.576+0000 7f1a22ffd640 10 client.?.objecter ms_handle_connect 0x7f1a2c142a20 2024-03-25T18:24:28.578 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.576+0000 7f1a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340653767 0 0) 0x7f1a2c13b110 con 0x7f1a2c142a20 2024-03-25T18:24:28.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.605 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:24:28.606 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.604+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a0c004eb0 con 0x7f1a2c142a20 2024-03-25T18:24:28.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686763092 0 0) 0x7f1a2c0abc90 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.608 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.609 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.609 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:24:28.609 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.607+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c13b110 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560434766 0 0) 0x7f1a2c1dbe60 con 0x7f1a2c0b4700 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:24:28.611 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.610+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c0abc90 con 0x7f1a2c0b4700 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926316718 0 0) 0x7f1a0c004eb0 con 0x7f1a2c142a20 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2c1dbe60 con 0x7f1a2c142a20 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a24001e80 con 0x7f1a2c142a20 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.612 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.613 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.611+0000 7f1a22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:28.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480564599 0 0) 0x7f1a2c13b110 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a0c004eb0 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a1c0019f0 con 0x7f1a2c0a96b0 2024-03-25T18:24:28.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.615 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.614+0000 7f1a22ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.616 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.615+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 539310168 0 0) 0x7f1a2c0abc90 con 0x7f1a2c0b4700 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.616+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2c13b110 con 0x7f1a2c0b4700 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.616+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a14001930 con 0x7f1a2c0b4700 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.616+0000 7f1a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.616+0000 7f1a22ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.616+0000 7f1a22ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:28.617 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.618 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 404546417 0 0) 0x7f1a2c1dbe60 con 0x7f1a2c142a20 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 >> v1:172.21.15.176:6790/0 conn(0x7f1a2c0a96b0 legacy=0x7f1a2c1db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 >> v1:172.21.15.186:6789/0 conn(0x7f1a2c0b4700 legacy=0x7f1a2c1397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 1 monclient: found mon.a 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0abf70 con 0x7f1a2c142a20 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 10 monclient: _renew_subs 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c0ac250 con 0x7f1a2c142a20 2024-03-25T18:24:28.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 10 monclient: _renew_subs 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c14e6c0 con 0x7f1a2c142a20 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.617+0000 7f1a308b3640 1 librados: init done 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.619+0000 7f1a308b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.619+0000 7f1a308b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.619+0000 7f1a308b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.619+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c04f730 con 0x7f1a2c142a20 2024-03-25T18:24:28.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a24002020 con 0x7f1a2c142a20 2024-03-25T18:24:28.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1a24004fb0 con 0x7f1a2c142a20 2024-03-25T18:24:28.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 10 monclient: dump: 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a2959a640 4 set_mon_vals no callback set 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1a240054a0 con 0x7f1a2c142a20 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1879976410 0 0) 0x7f1a24006a00 con 0x7f1a2c142a20 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 10 client.4131.objecter ms_dispatch 0x7f1a2c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:24:28.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:24:28.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:24:28.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.621+0000 7f1a22ffd640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:24:28.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.718+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1a24006690 con 0x7f1a2c142a20 2024-03-25T18:24:28.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.718+0000 7f1a22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.718+0000 7f1a22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:28.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.791+0000 7f1a308b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "62facedd-b505-4f2b-a30a-86f1e7366d4f", "id": 2}] 2024-03-25T18:24:28.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.791+0000 7f1a308b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "62facedd-b505-4f2b-a30a-86f1e7366d4f", "id": 2}] 2024-03-25T18:24:28.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.791+0000 7f1a308b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.791+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "62facedd-b505-4f2b-a30a-86f1e7366d4f", "id": 2} v 0) v1 -- 0x7f1a2c13b2a0 con 0x7f1a2c142a20 2024-03-25T18:24:28.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.823+0000 7f1a22ffd640 1 -- 172.21.15.186:0/1338857370 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "62facedd-b505-4f2b-a30a-86f1e7366d4f", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3594126733 0 216599125) 0x7f1a24005750 con 0x7f1a2c142a20 2024-03-25T18:24:28.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "62facedd-b505-4f2b-a30a-86f1e7366d4f", "id": 2}] 2024-03-25T18:24:28.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:28.825 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-03-25T18:24:28.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a308b3640 10 librados: watch_flush enter 2024-03-25T18:24:28.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a308b3640 10 librados: watch_flush exit 2024-03-25T18:24:28.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a308b3640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:24:28.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.824+0000 7f1a308b3640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 10 monclient: shutdown 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 >> v1:172.21.15.176:6789/0 conn(0x7f1a2c142a20 legacy=0x7f1a2c1db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 shutdown_connections 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 >> 172.21.15.186:0/1338857370 conn(0x7f1a2c0668e0 msgr2=0x7f1a2c13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 shutdown_connections 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 -- 172.21.15.186:0/1338857370 wait complete. 2024-03-25T18:24:28.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.825+0000 7f1a308b3640 1 librados: shutdown 2024-03-25T18:24:28.829 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.828+0000 7fc9c5a40040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:24:28.838 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 81d20a92-974e-43cb-b651-9aeb2a692a3b 3 2024-03-25T18:24:28.875 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:28.874+0000 7f9d251b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:24:28.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.937+0000 7fed7a9c6640 10 monclient: get_monmap_and_config 2024-03-25T18:24:28.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.937+0000 7fed7a9c6640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.937+0000 7fed7a9c6640 10 monclient: monmap: 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:28.938727+0000 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:28.938727+0000 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:28.939 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.938+0000 7fed7a9c6640 1 Processor -- start 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.938+0000 7fed7a9c6640 1 -- start start 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.938+0000 7fed7a9c6640 10 monclient: init 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-a con 0x7fed740b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-b con 0x7fed74142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-c con 0x7fed740a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740a9d60 con 0x7fed740b4700 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7413ff10 con 0x7fed740a96b0 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740aa1f0 con 0x7fed74142a20 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed7a9c6640 10 monclient(hunting): authenticate will time out at 8145.324570s 2024-03-25T18:24:28.941 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed78f63640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fed740a96b0 0x7fed7413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39282/0 (socket says 172.21.15.186:39282) 2024-03-25T18:24:28.942 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.939+0000 7fed78f63640 1 -- 172.21.15.186:0/1710604609 learned_addr learned my addr 172.21.15.186:0/1710604609 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785445869 0 0) 0x7fed7413ff10 con 0x7fed740a96b0 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.941+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c003520 con 0x7fed740a96b0 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239056642 0 0) 0x7fed740a9d60 con 0x7fed740b4700 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:24:28.943 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7413ff10 con 0x7fed740b4700 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005603083 0 0) 0x7fed740aa1f0 con 0x7fed74142a20 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:24:28.944 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.942+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed740a9d60 con 0x7fed74142a20 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565931797 0 0) 0x7fed7413ff10 con 0x7fed740b4700 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 >> v1:172.21.15.176:6790/0 conn(0x7fed74142a20 legacy=0x7fed7413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 >> v1:172.21.15.186:6789/0 conn(0x7fed740a96b0 legacy=0x7fed7413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 monclient: found mon.noname-a 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740b4ae0 con 0x7fed740b4700 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed68001670 con 0x7fed740b4700 2024-03-25T18:24:28.947 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed7a9c6640 5 monclient: authenticate success, global_id 4134 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed7a9c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed72ffd640 10 monclient: dump: 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.948 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.946+0000 7fed7a9c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed680017d0 con 0x7fed740b4700 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 1 -- 172.21.15.186:0/1710604609 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed68001be0 con 0x7fed740b4700 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed7a9c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed7a9c6640 4 set_mon_vals no callback set 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.947+0000 7fed72ffd640 10 monclient: dump: 2024-03-25T18:24:28.950 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 10 monclient: shutdown 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 1 -- 172.21.15.186:0/1710604609 >> v1:172.21.15.176:6789/0 conn(0x7fed740b4700 legacy=0x7fed74142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 1 -- 172.21.15.186:0/1710604609 shutdown_connections 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 1 -- 172.21.15.186:0/1710604609 >> 172.21.15.186:0/1710604609 conn(0x7fed740668e0 msgr2=0x7fed74066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 1 -- 172.21.15.186:0/1710604609 shutdown_connections 2024-03-25T18:24:28.951 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.948+0000 7fed7a9c6640 1 -- 172.21.15.186:0/1710604609 wait complete. 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.949+0000 7fed7a9c6640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.949+0000 7fed7a9c6640 10 monclient: monmap: 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.949+0000 7fed7a9c6640 1 librados: starting msgr at 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.949+0000 7fed7a9c6640 1 librados: starting objecter 2024-03-25T18:24:28.952 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.949+0000 7fed7a9c6640 1 Processor -- start 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 -- start start 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 librados: setting wanted keys 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 librados: calling monclient init 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient: init 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-b con 0x7fed740a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-a con 0x7fed740b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient(hunting): picked mon.noname-c con 0x7fed74142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7413a3a0 con 0x7fed740b4700 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7413b420 con 0x7fed740a96b0 2024-03-25T18:24:28.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed741dbe60 con 0x7fed74142a20 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.950+0000 7fed7a9c6640 10 monclient(hunting): authenticate will time out at 8145.335954s 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed73fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fed740a96b0 0x7fed7413ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39290/0 (socket says 172.21.15.186:39290) 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed73fff640 1 -- 172.21.15.186:0/2572947199 learned_addr learned my addr 172.21.15.186:0/2572947199 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter ms_handle_connect 0x7fed740a96b0 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter ms_handle_connect 0x7fed740b4700 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter ms_handle_connect 0x7fed74142a20 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.951+0000 7fed70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573106441 0 0) 0x7fed7413b420 con 0x7fed740a96b0 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed60004eb0 con 0x7fed740a96b0 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551322676 0 0) 0x7fed7413a3a0 con 0x7fed740b4700 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.953+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7413b420 con 0x7fed740b4700 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564967638 0 0) 0x7fed741dbe60 con 0x7fed74142a20 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:24:28.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.954+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7413a3a0 con 0x7fed74142a20 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.958+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 427770047 0 0) 0x7fed7413b420 con 0x7fed740b4700 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed741dbe60 con 0x7fed740b4700 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed64001870 con 0x7fed740b4700 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.960 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293878659 0 0) 0x7fed60004eb0 con 0x7fed740a96b0 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7413b420 con 0x7fed740a96b0 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed68001670 con 0x7fed740a96b0 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.961 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.962 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.962 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.962 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76877015 0 0) 0x7fed7413a3a0 con 0x7fed74142a20 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed60004eb0 con 0x7fed74142a20 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed6c001a30 con 0x7fed74142a20 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.959+0000 7fed70ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:28.963 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025517873 0 0) 0x7fed741dbe60 con 0x7fed740b4700 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 >> v1:172.21.15.176:6790/0 conn(0x7fed74142a20 legacy=0x7fed741db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 >> v1:172.21.15.186:6789/0 conn(0x7fed740a96b0 legacy=0x7fed7413ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 1 monclient: found mon.a 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7413a610 con 0x7fed740b4700 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 10 monclient: _renew_subs 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7413a950 con 0x7fed740b4700 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 10 monclient: _renew_subs 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed74138a00 con 0x7fed740b4700 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.961+0000 7fed7a9c6640 1 librados: init done 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.962+0000 7fed7a9c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.962+0000 7fed7a9c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.962+0000 7fed7a9c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:28.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:28.962+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed74138f50 con 0x7fed740b4700 2024-03-25T18:24:28.975 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:28.975+0000 7fc9c5a40040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:24:29.015 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.014+0000 7f9d251b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed64003610 con 0x7fed740b4700 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed64001d80 con 0x7fed740b4700 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 10 monclient: dump: 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.061 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed64003d30 con 0x7fed740b4700 2024-03-25T18:24:29.062 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.060+0000 7fed727fc640 4 set_mon_vals no callback set 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1719969224 0 0) 0x7fed64005440 con 0x7fed740b4700 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 10 client.4137.objecter ms_dispatch 0x7fed74000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed64004fb0 con 0x7fed740b4700 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.063 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.061+0000 7fed70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:29.106 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.105+0000 7fc9c5a40040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:24:29.126 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.125+0000 7f9d251b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:24:29.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.135+0000 7fed7a9c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "81d20a92-974e-43cb-b651-9aeb2a692a3b", "id": 3}] 2024-03-25T18:24:29.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.135+0000 7fed7a9c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "81d20a92-974e-43cb-b651-9aeb2a692a3b", "id": 3}] 2024-03-25T18:24:29.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.135+0000 7fed7a9c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:29.136 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.135+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "81d20a92-974e-43cb-b651-9aeb2a692a3b", "id": 3} v 0) v1 -- 0x7fed7404f730 con 0x7fed740b4700 2024-03-25T18:24:29.153 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.152+0000 7fc9c5a40040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:24:29.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.171+0000 7fed70ff9640 1 -- 172.21.15.186:0/2572947199 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81d20a92-974e-43cb-b651-9aeb2a692a3b", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 129216893 0 525046306) 0x7fed64005730 con 0x7fed740b4700 2024-03-25T18:24:29.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.171+0000 7fed70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "81d20a92-974e-43cb-b651-9aeb2a692a3b", "id": 3}] 2024-03-25T18:24:29.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.171+0000 7fed70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:29.172 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 10 librados: watch_flush enter 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 10 librados: watch_flush exit 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 10 monclient: shutdown 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 >> v1:172.21.15.176:6789/0 conn(0x7fed740b4700 legacy=0x7fed741db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 shutdown_connections 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 >> 172.21.15.186:0/2572947199 conn(0x7fed740668e0 msgr2=0x7fed740b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 shutdown_connections 2024-03-25T18:24:29.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.172+0000 7fed7a9c6640 1 -- 172.21.15.186:0/2572947199 wait complete. 2024-03-25T18:24:29.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.173+0000 7fed7a9c6640 1 librados: shutdown 2024-03-25T18:24:29.175 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.174+0000 7f9d251b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:24:29.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 4b761814-fb08-409b-ad5e-8161f0ab123a 4 2024-03-25T18:24:29.275 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.275+0000 7fc9c5a40040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.282+0000 7fdb57c6e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.282+0000 7fdb57c6e640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.282+0000 7fdb57c6e640 10 monclient: monmap: 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:29.283344+0000 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:29.283344+0000 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:29.283 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.282+0000 7fdb57c6e640 1 Processor -- start 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 1 -- start start 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient: init 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb500b4700 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb50142a20 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-a con 0x7fdb500a96b0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb500a9d60 con 0x7fdb500a96b0 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5013ff10 con 0x7fdb50142a20 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb500aa1f0 con 0x7fdb500b4700 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.283+0000 7fdb57c6e640 10 monclient(hunting): authenticate will time out at 8145.669214s 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.284+0000 7fdb55209640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdb50142a20 0x7fdb5013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39298/0 (socket says 172.21.15.186:39298) 2024-03-25T18:24:29.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.284+0000 7fdb55209640 1 -- 172.21.15.186:0/498368581 learned_addr learned my addr 172.21.15.186:0/498368581 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653417178 0 0) 0x7fdb500a9d60 con 0x7fdb500a96b0 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.286+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c003520 con 0x7fdb500a96b0 2024-03-25T18:24:29.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886232026 0 0) 0x7fdb5013ff10 con 0x7fdb50142a20 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb500a9d60 con 0x7fdb50142a20 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652814820 0 0) 0x7fdb500aa1f0 con 0x7fdb500b4700 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:24:29.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.287+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5013ff10 con 0x7fdb500b4700 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4016785685 0 0) 0x7fdb500a9d60 con 0x7fdb50142a20 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 >> v1:172.21.15.176:6790/0 conn(0x7fdb500b4700 legacy=0x7fdb50142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 >> v1:172.21.15.176:6789/0 conn(0x7fdb500a96b0 legacy=0x7fdb5013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 monclient: found mon.noname-c 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb500b4ae0 con 0x7fdb50142a20 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb4c001670 con 0x7fdb50142a20 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb54a08640 10 monclient: dump: 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.291 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb57c6e640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:24:29.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.289+0000 7fdb57c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:29.292 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.291+0000 7f9d251b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb4c0017d0 con 0x7fdb50142a20 2024-03-25T18:24:29.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 1 -- 172.21.15.186:0/498368581 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb4c001be0 con 0x7fdb50142a20 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.292+0000 7fdb54a08640 10 monclient: dump: 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:29.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 4 set_mon_vals no callback set 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 10 monclient: shutdown 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 1 -- 172.21.15.186:0/498368581 >> v1:172.21.15.186:6789/0 conn(0x7fdb50142a20 legacy=0x7fdb5013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 1 -- 172.21.15.186:0/498368581 shutdown_connections 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 1 -- 172.21.15.186:0/498368581 >> 172.21.15.186:0/498368581 conn(0x7fdb500668e0 msgr2=0x7fdb50066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 1 -- 172.21.15.186:0/498368581 shutdown_connections 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.293+0000 7fdb57c6e640 1 -- 172.21.15.186:0/498368581 wait complete. 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 10 monclient: monmap: 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:29.297 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 librados: starting msgr at 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 librados: starting objecter 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 Processor -- start 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 -- start start 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 librados: setting wanted keys 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 1 librados: calling monclient init 2024-03-25T18:24:29.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.294+0000 7fdb57c6e640 10 monclient: init 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb500a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb500b4700 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient(hunting): picked mon.noname-a con 0x7fdb50142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb501daba0 con 0x7fdb50142a20 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb500acf70 con 0x7fdb500a96b0 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb501db110 con 0x7fdb500b4700 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.295+0000 7fdb57c6e640 10 monclient(hunting): authenticate will time out at 8145.680928s 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb55a0a640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdb500a96b0 0x7fdb500abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39308/0 (socket says 172.21.15.186:39308) 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb55a0a640 1 -- 172.21.15.186:0/3044999991 learned_addr learned my addr 172.21.15.186:0/3044999991 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter ms_handle_connect 0x7fdb500a96b0 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter ms_handle_connect 0x7fdb50142a20 2024-03-25T18:24:29.299 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter ms_handle_connect 0x7fdb500b4700 2024-03-25T18:24:29.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.296+0000 7fdb467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.300 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426437336 0 0) 0x7fdb501daba0 con 0x7fdb50142a20 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb34004eb0 con 0x7fdb50142a20 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468131102 0 0) 0x7fdb500acf70 con 0x7fdb500a96b0 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb501daba0 con 0x7fdb500a96b0 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865561543 0 0) 0x7fdb501db110 con 0x7fdb500b4700 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:24:29.302 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.298+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb500acf70 con 0x7fdb500b4700 2024-03-25T18:24:29.304 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 499545733 0 0) 0x7fdb501daba0 con 0x7fdb500a96b0 2024-03-25T18:24:29.304 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb501db110 con 0x7fdb500a96b0 2024-03-25T18:24:29.304 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb400019f0 con 0x7fdb500a96b0 2024-03-25T18:24:29.304 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): dump: 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817613010 0 0) 0x7fdb34004eb0 con 0x7fdb50142a20 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb501daba0 con 0x7fdb50142a20 2024-03-25T18:24:29.305 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb48001af0 con 0x7fdb50142a20 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): dump: 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739097346 0 0) 0x7fdb500acf70 con 0x7fdb500b4700 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb34004eb0 con 0x7fdb500b4700 2024-03-25T18:24:29.306 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb4c001ae0 con 0x7fdb500b4700 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.303+0000 7fdb467fc640 10 monclient(hunting): dump: 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.307 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.308 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 908155318 0 0) 0x7fdb501db110 con 0x7fdb500a96b0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 >> v1:172.21.15.176:6790/0 conn(0x7fdb500b4700 legacy=0x7fdb5013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 >> v1:172.21.15.176:6789/0 conn(0x7fdb50142a20 legacy=0x7fdb500ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 1 monclient: found mon.b 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb501db2a0 con 0x7fdb500a96b0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb57c6e640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb57c6e640 10 monclient: _renew_subs 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb57c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.306+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb501db4f0 con 0x7fdb500a96b0 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.307+0000 7fdb57c6e640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.307+0000 7fdb57c6e640 10 monclient: _renew_subs 2024-03-25T18:24:29.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.307+0000 7fdb57c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.307+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb501362d0 con 0x7fdb500a96b0 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.307+0000 7fdb57c6e640 1 librados: init done 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.308+0000 7fdb57c6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.308+0000 7fdb57c6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.308+0000 7fdb57c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.310 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.308+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb500b1dc0 con 0x7fdb500a96b0 2024-03-25T18:24:29.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.376+0000 7fc9c5a40040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:24:29.390 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.389+0000 7f9d251b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:24:29.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb40003610 con 0x7fdb500a96b0 2024-03-25T18:24:29.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdb400039a0 con 0x7fdb500a96b0 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: dump: 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.503 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdb40003f50 con 0x7fdb500a96b0 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb47fff640 4 set_mon_vals no callback set 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 888007768 0 0) 0x7fdb400067a0 con 0x7fdb500a96b0 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 client.4144.objecter ms_dispatch 0x7fdb50000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdb40006be0 con 0x7fdb500a96b0 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.501+0000 7fdb467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:29.545 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.544+0000 7f9d251b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:29.554 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.553+0000 7fc9c5a40040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:29.588 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.588+0000 7f9d251b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:29.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.592+0000 7fdb57c6e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4b761814-fb08-409b-ad5e-8161f0ab123a", "id": 4}] 2024-03-25T18:24:29.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.592+0000 7fdb57c6e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4b761814-fb08-409b-ad5e-8161f0ab123a", "id": 4}] 2024-03-25T18:24:29.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.592+0000 7fdb57c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:29.593 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.592+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4b761814-fb08-409b-ad5e-8161f0ab123a", "id": 4} v 0) v1 -- 0x7fdb501dad90 con 0x7fdb500a96b0 2024-03-25T18:24:29.597 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.597+0000 7fc9c5a40040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:29.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.722+0000 7fdb467fc640 1 -- 172.21.15.186:0/3044999991 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b761814-fb08-409b-ad5e-8161f0ab123a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3559769623 0 1697012583) 0x7fdb400042d0 con 0x7fdb500a96b0 2024-03-25T18:24:29.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.722+0000 7fdb467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4b761814-fb08-409b-ad5e-8161f0ab123a", "id": 4}] 2024-03-25T18:24:29.723 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.722+0000 7fdb467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:29.723 INFO:teuthology.orchestra.run.smithi186.stdout:4 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 10 librados: watch_flush enter 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 10 librados: watch_flush exit 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 10 monclient: shutdown 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 >> v1:172.21.15.186:6789/0 conn(0x7fdb500a96b0 legacy=0x7fdb500abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 shutdown_connections 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 >> 172.21.15.186:0/3044999991 conn(0x7fdb500668e0 msgr2=0x7fdb5013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 shutdown_connections 2024-03-25T18:24:29.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 -- 172.21.15.186:0/3044999991 wait complete. 2024-03-25T18:24:29.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.723+0000 7fdb57c6e640 1 librados: shutdown 2024-03-25T18:24:29.734 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 5d991ea7-a0d8-4dab-95b4-8b314ec586d6 5 2024-03-25T18:24:29.764 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:29.762+0000 7fc9c5a40040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:24:29.769 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:29.769+0000 7f9d251b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.833+0000 7fd5aadaf640 10 monclient: get_monmap_and_config 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.833+0000 7fd5aadaf640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.833+0000 7fd5aadaf640 10 monclient: monmap: 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:29.834341+0000 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:29.834341+0000 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.834 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:29.835 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:29.835 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:29.835 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 1 Processor -- start 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 1 -- start start 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient: init 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a40b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a4142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a40a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a40a9d60 con 0x7fd5a40b4700 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a413ff10 con 0x7fd5a40a96b0 2024-03-25T18:24:29.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a40aa1f0 con 0x7fd5a4142a20 2024-03-25T18:24:29.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.834+0000 7fd5aadaf640 10 monclient(hunting): authenticate will time out at 8146.220137s 2024-03-25T18:24:29.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.835+0000 7fd5a3fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fd5a4142a20 0x7fd5a413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44282/0 (socket says 172.21.15.186:44282) 2024-03-25T18:24:29.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.835+0000 7fd5a3fff640 1 -- 172.21.15.186:0/1449712409 learned_addr learned my addr 172.21.15.186:0/1449712409 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097214269 0 0) 0x7fd5a40aa1f0 con 0x7fd5a4142a20 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c003520 con 0x7fd5a4142a20 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982837537 0 0) 0x7fd5a413ff10 con 0x7fd5a40a96b0 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.837+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a40aa1f0 con 0x7fd5a40a96b0 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629310337 0 0) 0x7fd5a40a9d60 con 0x7fd5a40b4700 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:24:29.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.838+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a413ff10 con 0x7fd5a40b4700 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075108496 0 0) 0x7fd58c003520 con 0x7fd5a4142a20 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 >> v1:172.21.15.186:6789/0 conn(0x7fd5a40a96b0 legacy=0x7fd5a413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 >> v1:172.21.15.176:6789/0 conn(0x7fd5a40b4700 legacy=0x7fd5a4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 monclient: found mon.noname-b 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40b4ae0 con 0x7fd5a4142a20 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd598001690 con 0x7fd5a4142a20 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5a37fe640 10 monclient: dump: 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.843 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5aadaf640 5 monclient: authenticate success, global_id 4148 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.841+0000 7fd5aadaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5980017f0 con 0x7fd5a4142a20 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 1 -- 172.21.15.186:0/1449712409 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd598001c00 con 0x7fd5a4142a20 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5a37fe640 10 monclient: dump: 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5aadaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:29.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.843+0000 7fd5aadaf640 4 set_mon_vals no callback set 2024-03-25T18:24:29.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 10 monclient: shutdown 2024-03-25T18:24:29.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 1 -- 172.21.15.186:0/1449712409 >> v1:172.21.15.176:6790/0 conn(0x7fd5a4142a20 legacy=0x7fd5a413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 1 -- 172.21.15.186:0/1449712409 shutdown_connections 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 1 -- 172.21.15.186:0/1449712409 >> 172.21.15.186:0/1449712409 conn(0x7fd5a40668e0 msgr2=0x7fd5a4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 1 -- 172.21.15.186:0/1449712409 shutdown_connections 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.844+0000 7fd5aadaf640 1 -- 172.21.15.186:0/1449712409 wait complete. 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 10 monclient: monmap: 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:29.848 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 librados: starting msgr at 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 librados: starting objecter 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 Processor -- start 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 -- start start 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 librados: setting wanted keys 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 1 librados: calling monclient init 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.845+0000 7fd5aadaf640 10 monclient: init 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a40a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a40b4700 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a4142a20 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:29.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a41dab30 con 0x7fd5a40b4700 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a40ab880 con 0x7fd5a40a96b0 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a41d7510 con 0x7fd5a4142a20 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5aadaf640 10 monclient(hunting): authenticate will time out at 8146.231774s 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5a934c640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fd5a4142a20 0x7fd5a40ac340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44296/0 (socket says 172.21.15.186:44296) 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.846+0000 7fd5a934c640 1 -- 172.21.15.186:0/827409062 learned_addr learned my addr 172.21.15.186:0/827409062 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter ms_handle_connect 0x7fd5a4142a20 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter ms_handle_connect 0x7fd5a40b4700 2024-03-25T18:24:29.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter ms_handle_connect 0x7fd5a40a96b0 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.847+0000 7fd5a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884456316 0 0) 0x7fd5a41d7510 con 0x7fd5a4142a20 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd590004eb0 con 0x7fd5a4142a20 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324863659 0 0) 0x7fd5a41dab30 con 0x7fd5a40b4700 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a41d7510 con 0x7fd5a40b4700 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383171617 0 0) 0x7fd5a40ab880 con 0x7fd5a40a96b0 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:24:29.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.849+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a41dab30 con 0x7fd5a40a96b0 2024-03-25T18:24:29.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2002788668 0 0) 0x7fd590004eb0 con 0x7fd5a4142a20 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a40ab880 con 0x7fd5a4142a20 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd594001910 con 0x7fd5a4142a20 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 10 monclient(hunting): dump: 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.855 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.853+0000 7fd5a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985454386 0 0) 0x7fd5a41d7510 con 0x7fd5a40b4700 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd590004eb0 con 0x7fd5a40b4700 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd598001b00 con 0x7fd5a40b4700 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): dump: 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.856 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.857 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.857 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.857 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233606287 0 0) 0x7fd5a41dab30 con 0x7fd5a40a96b0 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a41d7510 con 0x7fd5a40a96b0 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd59c001bb0 con 0x7fd5a40a96b0 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.854+0000 7fd5a17fa640 10 monclient(hunting): dump: 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:29.859 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 238972072 0 0) 0x7fd5a40ab880 con 0x7fd5a4142a20 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 >> v1:172.21.15.186:6789/0 conn(0x7fd5a40a96b0 legacy=0x7fd5a41d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 >> v1:172.21.15.176:6789/0 conn(0x7fd5a40b4700 legacy=0x7fd5a41db050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 1 monclient: found mon.c 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a41d76a0 con 0x7fd5a4142a20 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 5 monclient: authenticate success, global_id 4151 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 10 monclient: _renew_subs 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a41d8280 con 0x7fd5a4142a20 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 10 monclient: _renew_subs 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a41db940 con 0x7fd5a4142a20 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.856+0000 7fd5aadaf640 1 librados: init done 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.857+0000 7fd5aadaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.857+0000 7fd5aadaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.857+0000 7fd5aadaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:29.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:29.857+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a40ae0a0 con 0x7fd5a4142a20 2024-03-25T18:24:30.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5940035f0 con 0x7fd5a4142a20 2024-03-25T18:24:30.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd594003940 con 0x7fd5a4142a20 2024-03-25T18:24:30.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 10 monclient: dump: 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd594003ef0 con 0x7fd5a4142a20 2024-03-25T18:24:30.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.038+0000 7fd5a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.040+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2487178258 0 0) 0x7fd594006710 con 0x7fd5a4142a20 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.040+0000 7fd5a17fa640 10 client.4151.objecter ms_dispatch 0x7fd5a4000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.040+0000 7fd5a17fa640 3 client.4151.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.040+0000 7fd5a17fa640 3 client.4151.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.040+0000 7fd5a17fa640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.041+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd594004420 con 0x7fd5a4142a20 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.041+0000 7fd5a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.041+0000 7fd5a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:30.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.112+0000 7fd5aadaf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5d991ea7-a0d8-4dab-95b4-8b314ec586d6", "id": 5}] 2024-03-25T18:24:30.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.112+0000 7fd5aadaf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5d991ea7-a0d8-4dab-95b4-8b314ec586d6", "id": 5}] 2024-03-25T18:24:30.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.112+0000 7fd5aadaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.112+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5d991ea7-a0d8-4dab-95b4-8b314ec586d6", "id": 5} v 0) v1 -- 0x7fd5a404f730 con 0x7fd5a4142a20 2024-03-25T18:24:30.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.214+0000 7fd5a17fa640 1 -- 172.21.15.186:0/827409062 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d991ea7-a0d8-4dab-95b4-8b314ec586d6", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3773450357 0 1988412176) 0x7fd594006190 con 0x7fd5a4142a20 2024-03-25T18:24:30.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.214+0000 7fd5a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5d991ea7-a0d8-4dab-95b4-8b314ec586d6", "id": 5}] 2024-03-25T18:24:30.215 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.214+0000 7fd5a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:30.216 INFO:teuthology.orchestra.run.smithi186.stdout:5 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 10 librados: watch_flush enter 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 10 librados: watch_flush exit 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 10 client.4151.objecter successfully canceled tick 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 10 monclient: shutdown 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 >> v1:172.21.15.176:6790/0 conn(0x7fd5a4142a20 legacy=0x7fd5a40ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 shutdown_connections 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.215+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 >> 172.21.15.186:0/827409062 conn(0x7fd5a40668e0 msgr2=0x7fd5a4066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.216+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 shutdown_connections 2024-03-25T18:24:30.217 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.216+0000 7fd5aadaf640 1 -- 172.21.15.186:0/827409062 wait complete. 2024-03-25T18:24:30.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.216+0000 7fd5aadaf640 1 librados: shutdown 2024-03-25T18:24:30.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new f87e7ad2-713e-483d-9d9d-1329acb84493 6 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 10 monclient: get_monmap_and_config 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 10 monclient: monmap: 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:30.367121+0000 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:30.367121+0000 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:30.367 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 1 Processor -- start 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 1 -- start start 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.366+0000 7fdda6a9a640 10 monclient: init 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fdda00a9710 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fdda00aafa0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fdda013c430 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda00ab450 con 0x7fdda00a9710 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda00abdb0 con 0x7fdda00aafa0 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda00abf40 con 0x7fdda013c430 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.367+0000 7fdda6a9a640 10 monclient(hunting): authenticate will time out at 8146.752949s 2024-03-25T18:24:30.369 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.368+0000 7fdd9ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdda00aafa0 0x7fdda00ad3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:39344/0 (socket says 172.21.15.186:39344) 2024-03-25T18:24:30.370 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.368+0000 7fdd9ffff640 1 -- 172.21.15.186:0/4057403095 learned_addr learned my addr 172.21.15.186:0/4057403095 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965640788 0 0) 0x7fdda00abf40 con 0x7fdda013c430 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd88003520 con 0x7fdda013c430 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609293982 0 0) 0x7fdda00abdb0 con 0x7fdda00aafa0 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:24:30.371 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda00abf40 con 0x7fdda00aafa0 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048628239 0 0) 0x7fdda00ab450 con 0x7fdda00a9710 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:24:30.372 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.370+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda00abdb0 con 0x7fdda00a9710 2024-03-25T18:24:30.374 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235081522 0 0) 0x7fdda00abf40 con 0x7fdda00aafa0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 >> v1:172.21.15.176:6790/0 conn(0x7fdda013c430 legacy=0x7fdda014a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 >> v1:172.21.15.176:6789/0 conn(0x7fdda00a9710 legacy=0x7fdda0072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda013c860 con 0x7fdda00aafa0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd94001670 con 0x7fdda00aafa0 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdd9f7fe640 10 monclient: dump: 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.375 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdda6a9a640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:24:30.376 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.373+0000 7fdda6a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd94001850 con 0x7fdda00aafa0 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 1 -- 172.21.15.186:0/4057403095 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd94001c60 con 0x7fdda00aafa0 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdd9f7fe640 10 monclient: dump: 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.378 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdda6a9a640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.375+0000 7fdda6a9a640 4 set_mon_vals no callback set 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 10 monclient: shutdown 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 1 -- 172.21.15.186:0/4057403095 >> v1:172.21.15.186:6789/0 conn(0x7fdda00aafa0 legacy=0x7fdda00ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 1 -- 172.21.15.186:0/4057403095 shutdown_connections 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 1 -- 172.21.15.186:0/4057403095 >> 172.21.15.186:0/4057403095 conn(0x7fdda006b560 msgr2=0x7fdda00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 1 -- 172.21.15.186:0/4057403095 shutdown_connections 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 1 -- 172.21.15.186:0/4057403095 wait complete. 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.376+0000 7fdda6a9a640 10 monclient: monmap: 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 librados: starting msgr at 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 librados: starting objecter 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 Processor -- start 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 -- start start 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 librados: setting wanted keys 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 1 librados: calling monclient init 2024-03-25T18:24:30.380 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.377+0000 7fdda6a9a640 10 monclient: init 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-c con 0x7fdda00a9710 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-b con 0x7fdda00aafa0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient(hunting): picked mon.noname-a con 0x7fdda013c430 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda01d7370 con 0x7fdda013c430 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda011c470 con 0x7fdda00aafa0 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda01d75a0 con 0x7fdda00a9710 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda6a9a640 10 monclient(hunting): authenticate will time out at 8146.763487s 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda4836640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fdda00a9710 0x7fdda01d6c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44310/0 (socket says 172.21.15.186:44310) 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdda4836640 1 -- 172.21.15.186:0/1825744907 learned_addr learned my addr 172.21.15.186:0/1825744907 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:30.383 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdd9d7fa640 10 client.?.objecter ms_handle_connect 0x7fdda00aafa0 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdd9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdd9d7fa640 10 client.?.objecter ms_handle_connect 0x7fdda00a9710 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.378+0000 7fdd9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.379+0000 7fdd9d7fa640 10 client.?.objecter ms_handle_connect 0x7fdda013c430 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.379+0000 7fdd9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096328847 0 0) 0x7fdda01d75a0 con 0x7fdda00a9710 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd8c004eb0 con 0x7fdda00a9710 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969323709 0 0) 0x7fdda011c470 con 0x7fdda00aafa0 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.380+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda01d75a0 con 0x7fdda00aafa0 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391614276 0 0) 0x7fdda01d7370 con 0x7fdda013c430 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:24:30.385 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.381+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda011c470 con 0x7fdda013c430 2024-03-25T18:24:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2001305154 0 0) 0x7fdda01d75a0 con 0x7fdda00aafa0 2024-03-25T18:24:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda01d7370 con 0x7fdda00aafa0 2024-03-25T18:24:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd94001b60 con 0x7fdda00aafa0 2024-03-25T18:24:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.386 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): dump: 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674196359 0 0) 0x7fdd8c004eb0 con 0x7fdda00a9710 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda01d75a0 con 0x7fdda00a9710 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd90001a10 con 0x7fdda00a9710 2024-03-25T18:24:30.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.385+0000 7fdd9d7fa640 10 monclient(hunting): dump: 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.388 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 902341023 0 0) 0x7fdda01d7370 con 0x7fdda00aafa0 2024-03-25T18:24:30.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 >> v1:172.21.15.176:6790/0 conn(0x7fdda00a9710 legacy=0x7fdda01d6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 >> v1:172.21.15.176:6789/0 conn(0x7fdda013c430 legacy=0x7fdda0070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 1 monclient: found mon.b 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda01378e0 con 0x7fdda00aafa0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdd9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 10 monclient: _renew_subs 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda0136e40 con 0x7fdda00aafa0 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 10 monclient: _renew_subs 2024-03-25T18:24:30.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda0136860 con 0x7fdda00aafa0 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.388+0000 7fdda6a9a640 1 librados: init done 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.389+0000 7fdda6a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.389+0000 7fdda6a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.389+0000 7fdda6a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.389+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda00acda0 con 0x7fdda00aafa0 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd94001d20 con 0x7fdda00aafa0 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd94004d90 con 0x7fdda00aafa0 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:30.486 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: dump: 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdd94005280 con 0x7fdda00aafa0 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4133071516 0 0) 0x7fdd94006970 con 0x7fdda00aafa0 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9effd640 4 set_mon_vals no callback set 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 client.4156.objecter ms_dispatch 0x7fdda0000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:24:30.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:24:30.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:24:30.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdd94018de0 con 0x7fdda00aafa0 2024-03-25T18:24:30.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.485+0000 7fdd9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:30.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.557+0000 7fdda6a9a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f87e7ad2-713e-483d-9d9d-1329acb84493", "id": 6}] 2024-03-25T18:24:30.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.557+0000 7fdda6a9a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f87e7ad2-713e-483d-9d9d-1329acb84493", "id": 6}] 2024-03-25T18:24:30.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.557+0000 7fdda6a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:30.559 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.557+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f87e7ad2-713e-483d-9d9d-1329acb84493", "id": 6} v 0) v1 -- 0x7fdda01d7730 con 0x7fdda00aafa0 2024-03-25T18:24:30.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.622+0000 7fdd9d7fa640 1 -- 172.21.15.186:0/1825744907 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f87e7ad2-713e-483d-9d9d-1329acb84493", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1840522213 0 1113811849) 0x7fdd94006380 con 0x7fdda00aafa0 2024-03-25T18:24:30.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.622+0000 7fdd9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f87e7ad2-713e-483d-9d9d-1329acb84493", "id": 6}] 2024-03-25T18:24:30.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.622+0000 7fdd9d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:30.624 INFO:teuthology.orchestra.run.smithi186.stdout:6 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 10 librados: watch_flush enter 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 10 librados: watch_flush exit 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 10 monclient: shutdown 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 >> v1:172.21.15.186:6789/0 conn(0x7fdda00aafa0 legacy=0x7fdda0136150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 shutdown_connections 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.623+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 >> 172.21.15.186:0/1825744907 conn(0x7fdda006b560 msgr2=0x7fdda0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.624+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 shutdown_connections 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.624+0000 7fdda6a9a640 1 -- 172.21.15.186:0/1825744907 wait complete. 2024-03-25T18:24:30.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.624+0000 7fdda6a9a640 1 librados: shutdown 2024-03-25T18:24:30.627 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:30.626+0000 7f9d251b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:24:30.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 700760af-5407-4183-9f2b-1b503a38ccd7 7 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 10 monclient: monmap: 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:24:30.737971+0000 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:24:30.737971+0000 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.738 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 1 Processor -- start 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 1 -- start start 2024-03-25T18:24:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.737+0000 7f9b2cb0e640 10 monclient: init 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b280b4700 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b28142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b280a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280a9d60 con 0x7f9b28142a20 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2813ff10 con 0x7f9b280a96b0 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280aa1f0 con 0x7f9b280b4700 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2cb0e640 10 monclient(hunting): authenticate will time out at 8147.123767s 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2659c640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f9b280b4700 0x7f9b28142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44326/0 (socket says 172.21.15.186:44326) 2024-03-25T18:24:30.740 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.738+0000 7f9b2659c640 1 -- 172.21.15.186:0/290154386 learned_addr learned my addr 172.21.15.186:0/290154386 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156500246 0 0) 0x7f9b280aa1f0 con 0x7f9b280b4700 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:24:30.741 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.740+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b10003520 con 0x7f9b280b4700 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944169167 0 0) 0x7f9b280a9d60 con 0x7f9b28142a20 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.742+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b280aa1f0 con 0x7f9b28142a20 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776939587 0 0) 0x7f9b2813ff10 con 0x7f9b280a96b0 2024-03-25T18:24:30.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:24:30.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.743+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b280a9d60 con 0x7f9b280a96b0 2024-03-25T18:24:30.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270402145 0 0) 0x7f9b10003520 con 0x7f9b280b4700 2024-03-25T18:24:30.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 >> v1:172.21.15.186:6789/0 conn(0x7f9b280a96b0 legacy=0x7f9b2813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 >> v1:172.21.15.176:6789/0 conn(0x7f9b28142a20 legacy=0x7f9b2813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.745 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 monclient: found mon.noname-b 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b280b4ae0 con 0x7f9b280b4700 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b1c001690 con 0x7f9b280b4700 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2559a640 10 monclient: dump: 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.746 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2cb0e640 5 monclient: authenticate success, global_id 4160 2024-03-25T18:24:30.747 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.744+0000 7f9b2cb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:30.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b1c0017f0 con 0x7f9b280b4700 2024-03-25T18:24:30.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.748 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 1 -- 172.21.15.186:0/290154386 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b1c001c00 con 0x7f9b280b4700 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2cb0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2cb0e640 4 set_mon_vals no callback set 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.745+0000 7f9b2559a640 10 monclient: dump: 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.749 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 10 monclient: shutdown 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/290154386 >> v1:172.21.15.176:6790/0 conn(0x7f9b280b4700 legacy=0x7f9b28142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/290154386 shutdown_connections 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/290154386 >> 172.21.15.186:0/290154386 conn(0x7f9b280668e0 msgr2=0x7f9b28066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/290154386 shutdown_connections 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.746+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/290154386 wait complete. 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.747+0000 7f9b2cb0e640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.747+0000 7f9b2cb0e640 10 monclient: monmap: 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.750 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.747+0000 7f9b2cb0e640 1 librados: starting msgr at 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.747+0000 7f9b2cb0e640 1 librados: starting objecter 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 Processor -- start 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 -- start start 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 librados: setting wanted keys 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 librados: calling monclient init 2024-03-25T18:24:30.751 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient: init 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b280a96b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b280b4700 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b28142a20 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280ad760 con 0x7f9b28142a20 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280ad9d0 con 0x7f9b280a96b0 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2813ba60 con 0x7f9b280b4700 2024-03-25T18:24:30.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b2cb0e640 10 monclient(hunting): authenticate will time out at 8147.133965s 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b25d9b640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f9b280b4700 0x7f9b281d7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.186:44330/0 (socket says 172.21.15.186:44330) 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.748+0000 7f9b25d9b640 1 -- 172.21.15.186:0/3739536822 learned_addr learned my addr 172.21.15.186:0/3739536822 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.749+0000 7f9b16ffd640 10 client.?.objecter ms_handle_connect 0x7f9b280b4700 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.749+0000 7f9b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.749+0000 7f9b16ffd640 10 client.?.objecter ms_handle_connect 0x7f9b280a96b0 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.749+0000 7f9b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.750+0000 7f9b16ffd640 10 client.?.objecter ms_handle_connect 0x7f9b28142a20 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.750+0000 7f9b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466030439 0 0) 0x7f9b2813ba60 con 0x7f9b280b4700 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:24:30.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b08004eb0 con 0x7f9b280b4700 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32311432 0 0) 0x7f9b280ad9d0 con 0x7f9b280a96b0 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.751+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2813ba60 con 0x7f9b280a96b0 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805258831 0 0) 0x7f9b280ad760 con 0x7f9b28142a20 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:24:30.755 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.752+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b280ad9d0 con 0x7f9b28142a20 2024-03-25T18:24:30.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1919092074 0 0) 0x7f9b08004eb0 con 0x7f9b280b4700 2024-03-25T18:24:30.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b280ad760 con 0x7f9b280b4700 2024-03-25T18:24:30.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b18001990 con 0x7f9b280b4700 2024-03-25T18:24:30.756 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615488647 0 0) 0x7f9b2813ba60 con 0x7f9b280a96b0 2024-03-25T18:24:30.757 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b08004eb0 con 0x7f9b280a96b0 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b1c001b00 con 0x7f9b280a96b0 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.755+0000 7f9b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.758 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943146602 0 0) 0x7f9b280ad9d0 con 0x7f9b28142a20 2024-03-25T18:24:30.760 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b2813ba60 con 0x7f9b28142a20 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b20001c70 con 0x7f9b28142a20 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2153486519 0 0) 0x7f9b280ad760 con 0x7f9b280b4700 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 >> v1:172.21.15.186:6789/0 conn(0x7f9b280a96b0 legacy=0x7f9b280ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 >> v1:172.21.15.176:6789/0 conn(0x7f9b28142a20 legacy=0x7f9b28135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 monclient: found mon.c 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2813bbf0 con 0x7f9b280b4700 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b2cb0e640 5 monclient: authenticate success, global_id 4163 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b2cb0e640 10 monclient: _renew_subs 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b2cb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.758+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b281392f0 con 0x7f9b280b4700 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 monclient: _renew_subs 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.762 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b280ac5c0 con 0x7f9b280b4700 2024-03-25T18:24:30.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 1 librados: init done 2024-03-25T18:24:30.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.759+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2804f730 con 0x7f9b280b4700 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b180035f0 con 0x7f9b280b4700 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9b18003a00 con 0x7f9b280b4700 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 10 monclient: dump: 2024-03-25T18:24:30.764 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 3 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b24d99640 4 set_mon_vals no callback set 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b18003fb0 con 0x7f9b280b4700 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4162539651 0 0) 0x7f9b18006830 con 0x7f9b280b4700 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 10 client.4163.objecter ms_dispatch 0x7f9b28000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 3 client.4163.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 3 client.4163.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:24:30.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.763+0000 7f9b16ffd640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-25T18:24:30.842 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:30.841+0000 7fc9c5a40040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:24:30.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.856+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b18004260 con 0x7f9b280b4700 2024-03-25T18:24:30.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.856+0000 7f9b16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.857 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.856+0000 7f9b16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:30.922 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:30.922+0000 7f9d251b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:24:30.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.927+0000 7f9b2cb0e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "700760af-5407-4183-9f2b-1b503a38ccd7", "id": 7}] 2024-03-25T18:24:30.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.927+0000 7f9b2cb0e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "700760af-5407-4183-9f2b-1b503a38ccd7", "id": 7}] 2024-03-25T18:24:30.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.927+0000 7f9b2cb0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:30.928 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:30.927+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd new", "uuid": "700760af-5407-4183-9f2b-1b503a38ccd7", "id": 7} v 0) v1 -- 0x7f9b28069f30 con 0x7f9b280b4700 2024-03-25T18:24:30.984 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:30.983+0000 7f9d251b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:24:31.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.011+0000 7f9b16ffd640 1 -- 172.21.15.186:0/3739536822 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "700760af-5407-4183-9f2b-1b503a38ccd7", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1901771975 0 1371665406) 0x7f9b18006280 con 0x7f9b280b4700 2024-03-25T18:24:31.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.011+0000 7f9b16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "700760af-5407-4183-9f2b-1b503a38ccd7", "id": 7}] 2024-03-25T18:24:31.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.011+0000 7f9b16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:31.012 INFO:teuthology.orchestra.run.smithi186.stdout:7 2024-03-25T18:24:31.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 10 librados: watch_flush enter 2024-03-25T18:24:31.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 10 librados: watch_flush exit 2024-03-25T18:24:31.013 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 10 client.4163.objecter successfully canceled tick 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 10 monclient: shutdown 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 >> v1:172.21.15.176:6790/0 conn(0x7f9b280b4700 legacy=0x7f9b281d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 shutdown_connections 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 >> 172.21.15.186:0/3739536822 conn(0x7f9b280668e0 msgr2=0x7f9b28066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 shutdown_connections 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 -- 172.21.15.186:0/3739536822 wait complete. 2024-03-25T18:24:31.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-25T18:24:31.012+0000 7f9b2cb0e640 1 librados: shutdown 2024-03-25T18:24:31.023 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.023 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:24:31.023 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:24:31.026 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:24:31.026 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.026 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:24:31.026 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:24:31.028 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:24:31.028 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.029 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:24:31.029 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:24:31.031 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:24:31.031 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.031 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:24:31.031 DEBUG:teuthology.orchestra.run.smithi176:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:24:31.033 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:24:31.033 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.033 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:24:31.033 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:24:31.035 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:24:31.035 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.036 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:24:31.036 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:24:31.038 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:24:31.038 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.038 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:24:31.038 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:24:31.040 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:24:31.040 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:31.041 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:24:31.041 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:24:31.043 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:24:31.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:31.155 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:31.154+0000 7fc9c5a40040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:24:31.158 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:31.157+0000 7f9d251b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:24:31.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.187+0000 7ff4b4a55640 10 monclient: get_monmap_and_config 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.187+0000 7ff4b4a55640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.187+0000 7ff4b4a55640 10 monclient: monmap: 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:24:31.188198+0000 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:24:31.188198+0000 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:31.188 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.187+0000 7ff4b4a55640 1 Processor -- start 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 1 -- start start 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient: init 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b00a9920 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:31.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b45a0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b0146450 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.188+0000 7ff4b4a55640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006b100 con 0x7ff4b00b45a0 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4b4a55640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00b2940 con 0x7ff4b00a9920 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4b4a55640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b013fec0 con 0x7ff4b0146450 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4b4a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4b4a55640 10 monclient(hunting): authenticate will time out at 8149.500891s 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4aed9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7ff4b0146450 0x7ff4b014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48694/0 (socket says 172.21.15.176:48694) 2024-03-25T18:24:31.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.189+0000 7ff4aed9d640 1 -- 172.21.15.176:0/1894960917 learned_addr learned my addr 172.21.15.176:0/1894960917 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.191+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491033121 0 0) 0x7ff4b013fec0 con 0x7ff4b0146450 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.191+0000 7ff4ad59a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.191+0000 7ff4ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.191+0000 7ff4ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.191+0000 7ff4ad59a640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:24:31.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.192+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003520 con 0x7ff4b0146450 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309302028 0 0) 0x7ff4b00b2940 con 0x7ff4b00a9920 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:24:31.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.193+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b013fec0 con 0x7ff4b00a9920 2024-03-25T18:24:31.197 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548479240 0 0) 0x7ff498003520 con 0x7ff4b0146450 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 >> v1:172.21.15.186:6789/0 conn(0x7ff4b00a9920 legacy=0x7ff4b013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 >> v1:172.21.15.176:6789/0 conn(0x7ff4b00b45a0 legacy=0x7ff4b0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 monclient: found mon.noname-b 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0140050 con 0x7ff4b0146450 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff4a8001a10 con 0x7ff4b0146450 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4ad59a640 10 monclient: dump: 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4b4a55640 5 monclient: authenticate success, global_id 4166 2024-03-25T18:24:31.199 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.196+0000 7ff4b4a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a8001b70 con 0x7ff4b0146450 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 1 -- 172.21.15.176:0/1894960917 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff4a8001de0 con 0x7ff4b0146450 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4b4a55640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4b4a55640 4 set_mon_vals no callback set 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4ad59a640 10 monclient: dump: 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4b4a55640 10 monclient: shutdown 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4b4a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.199+0000 7ff4b4a55640 1 -- 172.21.15.176:0/1894960917 >> v1:172.21.15.176:6790/0 conn(0x7ff4b0146450 legacy=0x7ff4b014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.200+0000 7ff4b4a55640 1 -- 172.21.15.176:0/1894960917 shutdown_connections 2024-03-25T18:24:31.202 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.200+0000 7ff4b4a55640 1 -- 172.21.15.176:0/1894960917 >> 172.21.15.176:0/1894960917 conn(0x7ff4b00b7f40 msgr2=0x7ff4b00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.203 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.200+0000 7ff4b4a55640 1 -- 172.21.15.176:0/1894960917 shutdown_connections 2024-03-25T18:24:31.203 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.200+0000 7ff4b4a55640 1 -- 172.21.15.176:0/1894960917 wait complete. 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.201+0000 7ff4b4a55640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.201+0000 7ff4b4a55640 10 monclient: monmap: 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:31.205 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 librados: starting msgr at 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 librados: starting objecter 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 Processor -- start 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 -- start start 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 librados: setting wanted keys 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 1 librados: calling monclient init 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 10 monclient: init 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.202+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b00a9920 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b45a0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b0146450 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:31.206 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b01dfe50 con 0x7ff4b00b45a0 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b0137b30 con 0x7ff4b00a9920 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b01df930 con 0x7ff4b0146450 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4b4a55640 10 monclient(hunting): authenticate will time out at 8149.515002s 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4aed9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7ff4b0146450 0x7ff4b01df220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48704/0 (socket says 172.21.15.176:48704) 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.203+0000 7ff4aed9d640 1 -- 172.21.15.176:0/3705633306 learned_addr learned my addr 172.21.15.176:0/3705633306 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.204+0000 7ff49effd640 10 client.?.objecter ms_handle_connect 0x7ff4b0146450 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.204+0000 7ff49effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.204+0000 7ff49effd640 10 client.?.objecter ms_handle_connect 0x7ff4b00a9920 2024-03-25T18:24:31.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.204+0000 7ff49effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939776711 0 0) 0x7ff4b01df930 con 0x7ff4b0146450 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.206+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff490004eb0 con 0x7ff4b0146450 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368416392 0 0) 0x7ff4b0137b30 con 0x7ff4b00a9920 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 10 monclient(hunting): my global_id is 4168 2024-03-25T18:24:31.209 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.207+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b01df930 con 0x7ff4b00a9920 2024-03-25T18:24:31.212 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295838476 0 0) 0x7ff490004eb0 con 0x7ff4b0146450 2024-03-25T18:24:31.212 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b0137b30 con 0x7ff4b0146450 2024-03-25T18:24:31.212 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff4a8001f80 con 0x7ff4b0146450 2024-03-25T18:24:31.212 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.212 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 10 monclient(hunting): dump: 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.213 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.212+0000 7ff49effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:31.215 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 457373536 0 0) 0x7ff4b01df930 con 0x7ff4b00a9920 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff490004eb0 con 0x7ff4b00a9920 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff4a0001930 con 0x7ff4b00a9920 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.214+0000 7ff49effd640 10 monclient(hunting): dump: 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:31.216 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.215+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2265103214 0 0) 0x7ff4b0137b30 con 0x7ff4b0146450 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 >> v1:172.21.15.186:6789/0 conn(0x7ff4b00a9920 legacy=0x7ff4b013bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 >> v1:172.21.15.176:6789/0 conn(0x7ff4b00b45a0 legacy=0x7ff4b0137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 1 monclient: found mon.c 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b01dfac0 con 0x7ff4b0146450 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.219 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff49effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 5 monclient: authenticate success, global_id 4169 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 10 monclient: _renew_subs 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b01dbd20 con 0x7ff4b0146450 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 10 monclient: _renew_subs 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b01dc200 con 0x7ff4b0146450 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.216+0000 7ff4b4a55640 1 librados: init done 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.217+0000 7ff4b4a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.217+0000 7ff4b4a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.217+0000 7ff4b4a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.220 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.217+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b006ef10 con 0x7ff4b0146450 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a8004e10 con 0x7ff4b0146450 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff4a8005120 con 0x7ff4b0146450 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff4acd99640 4 set_mon_vals no callback set 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 10 monclient: dump: 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:31.223 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff4a8005610 con 0x7ff4b0146450 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7ff4a8006d00 con 0x7ff4b0146450 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 10 client.4169.objecter ms_dispatch 0x7ff4b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:31.224 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.221+0000 7ff49effd640 20 client.4169.objecter dump_active .. 0 homeless 2024-03-25T18:24:31.246 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:31.245+0000 7fc9c5a40040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:24:31.320 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.319+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff4a8006740 con 0x7ff4b0146450 2024-03-25T18:24:31.320 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.319+0000 7ff49effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.320 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.319+0000 7ff49effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:31.391 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.390+0000 7ff4b4a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.391 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.390+0000 7ff4b4a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.391 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.390+0000 7ff4b4a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:24:31.391 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.390+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b004f730 con 0x7ff4b0146450 2024-03-25T18:24:31.456 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-03-25T18:24:31.455+0000 7fc9c5a40040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:24:31.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.667+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1954661686 0 0) 0x7ff4a80069b0 con 0x7ff4b0146450 2024-03-25T18:24:31.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.737+0000 7ff49effd640 1 -- 172.21.15.176:0/3705633306 <== mon.2 v1:172.21.15.176:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7ff4a8016030 con 0x7ff4b0146450 2024-03-25T18:24:31.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.737+0000 7ff49effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.737+0000 7ff49effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:31.738 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-25T18:24:31.738 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:31.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 10 librados: watch_flush enter 2024-03-25T18:24:31.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 10 librados: watch_flush exit 2024-03-25T18:24:31.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 20 client.4169.objecter shutdown clearing up homeless session... 2024-03-25T18:24:31.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 10 client.4169.objecter successfully canceled tick 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 10 monclient: shutdown 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.738+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 >> v1:172.21.15.176:6790/0 conn(0x7ff4b0146450 legacy=0x7ff4b01df220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.739+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 shutdown_connections 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.739+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 >> 172.21.15.176:0/3705633306 conn(0x7ff4b00b7f40 msgr2=0x7ff4b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.739+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 shutdown_connections 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.739+0000 7ff4b4a55640 1 -- 172.21.15.176:0/3705633306 wait complete. 2024-03-25T18:24:31.740 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-25T18:24:31.739+0000 7ff4b4a55640 1 librados: shutdown 2024-03-25T18:24:31.754 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:24:31.755 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:24:32.560 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.706+0000 7fed8758c640 10 monclient: get_monmap_and_config 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.706+0000 7fed8758c640 10 monclient: build_initial_monmap 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.706+0000 7fed8758c640 10 monclient: monmap: 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:24:32.706918+0000 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:24:32.706918+0000 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:32.707 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:32.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.706+0000 7fed8758c640 1 Processor -- start 2024-03-25T18:24:32.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.708+0000 7fed8758c640 1 -- start start 2024-03-25T18:24:32.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.708+0000 7fed8758c640 10 monclient: init 2024-03-25T18:24:32.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:32.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-a con 0x7fed800a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-c con 0x7fed8013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-b con 0x7fed801430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed80142520 con 0x7fed800a98f0 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800b28a0 con 0x7fed8013f710 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800b2a30 con 0x7fed801430c0 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.709+0000 7fed8758c640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:32.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.710+0000 7fed8758c640 10 monclient(hunting): authenticate will time out at 8151.021851s 2024-03-25T18:24:32.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.775+0000 7fed85b29640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fed801430c0 0x7fed8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48774/0 (socket says 172.21.15.176:48774) 2024-03-25T18:24:32.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.775+0000 7fed85b29640 1 -- 172.21.15.176:0/2054817928 learned_addr learned my addr 172.21.15.176:0/2054817928 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367905176 0 0) 0x7fed800b28a0 con 0x7fed8013f710 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:24:32.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.988+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6c003520 con 0x7fed8013f710 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818369564 0 0) 0x7fed80142520 con 0x7fed800a98f0 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:24:32.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:32.997+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed800b28a0 con 0x7fed800a98f0 2024-03-25T18:24:33.040 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:33.039+0000 7f9cc6617640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:24:33.042 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:24:33.041+0000 7f9cc6617640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:24:33.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002805261 0 0) 0x7fed800b28a0 con 0x7fed800a98f0 2024-03-25T18:24:33.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 >> v1:172.21.15.176:6790/0 conn(0x7fed801430c0 legacy=0x7fed8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 >> v1:172.21.15.186:6789/0 conn(0x7fed8013f710 legacy=0x7fed801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 monclient: found mon.noname-a 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800b44c0 con 0x7fed800a98f0 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed7c001670 con 0x7fed800a98f0 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed77fff640 10 monclient: dump: 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.066 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed8758c640 5 monclient: authenticate success, global_id 4170 2024-03-25T18:24:33.067 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.064+0000 7fed8758c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed7c0017d0 con 0x7fed800a98f0 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed8758c640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed8758c640 4 set_mon_vals no callback set 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 1 -- 172.21.15.176:0/2054817928 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed7c001ba0 con 0x7fed800a98f0 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.202+0000 7fed77fff640 10 monclient: dump: 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.203 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:33.204 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:33.204 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:33.204 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:33.204 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 10 monclient: shutdown 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 1 -- 172.21.15.176:0/2054817928 >> v1:172.21.15.176:6789/0 conn(0x7fed800a98f0 legacy=0x7fed800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 1 -- 172.21.15.176:0/2054817928 shutdown_connections 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 1 -- 172.21.15.176:0/2054817928 >> 172.21.15.176:0/2054817928 conn(0x7fed800b7ea0 msgr2=0x7fed800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 1 -- 172.21.15.176:0/2054817928 shutdown_connections 2024-03-25T18:24:33.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.203+0000 7fed8758c640 1 -- 172.21.15.176:0/2054817928 wait complete. 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.204+0000 7fed8758c640 10 monclient: build_initial_monmap 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.204+0000 7fed8758c640 10 monclient: monmap: 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:24:33.206 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 librados: starting msgr at 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 librados: starting objecter 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 Processor -- start 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 -- start start 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 librados: setting wanted keys 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 1 librados: calling monclient init 2024-03-25T18:24:33.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.205+0000 7fed8758c640 10 monclient: init 2024-03-25T18:24:33.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.206+0000 7fed8758c640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.206+0000 7fed8758c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.206+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-a con 0x7fed800a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-b con 0x7fed8013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 10 monclient(hunting): picked mon.noname-c con 0x7fed801430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed801db410 con 0x7fed800a98f0 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800ab950 con 0x7fed8013f710 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed801db640 con 0x7fed801430c0 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.207+0000 7fed8758c640 10 monclient(hunting): authenticate will time out at 8151.519436s 2024-03-25T18:24:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.209+0000 7fed85b29640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fed801430c0 0x7fed801d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48780/0 (socket says 172.21.15.176:48780) 2024-03-25T18:24:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.209+0000 7fed85b29640 1 -- 172.21.15.176:0/2109939978 learned_addr learned my addr 172.21.15.176:0/2109939978 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.209+0000 7fed75ffb640 10 client.?.objecter ms_handle_connect 0x7fed801430c0 2024-03-25T18:24:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.209+0000 7fed75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:33.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.211+0000 7fed75ffb640 10 client.?.objecter ms_handle_connect 0x7fed800a98f0 2024-03-25T18:24:33.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.211+0000 7fed75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565911284 0 0) 0x7fed801db410 con 0x7fed800a98f0 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:24:33.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.215+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed64004eb0 con 0x7fed800a98f0 2024-03-25T18:24:33.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039458780 0 0) 0x7fed64004eb0 con 0x7fed800a98f0 2024-03-25T18:24:33.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed801db410 con 0x7fed800a98f0 2024-03-25T18:24:33.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed7c001ba0 con 0x7fed800a98f0 2024-03-25T18:24:33.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.223+0000 7fed75ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:33.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 771076079 0 0) 0x7fed801db410 con 0x7fed800a98f0 2024-03-25T18:24:33.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 >> v1:172.21.15.176:6790/0 conn(0x7fed801430c0 legacy=0x7fed801d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 >> v1:172.21.15.186:6789/0 conn(0x7fed8013f710 legacy=0x7fed800b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 1 monclient: found mon.a 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800b0df0 con 0x7fed800a98f0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.229+0000 7fed8758c640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 10 monclient: _renew_subs 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed800b1040 con 0x7fed800a98f0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 10 monclient: _renew_subs 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:33.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed800b09b0 con 0x7fed800a98f0 2024-03-25T18:24:33.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.230+0000 7fed8758c640 1 librados: init done 2024-03-25T18:24:33.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.231+0000 7fed8758c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.231+0000 7fed8758c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.231+0000 7fed8758c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:33.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.231+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed800b0c00 con 0x7fed800a98f0 2024-03-25T18:24:33.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.444+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed7c0038a0 con 0x7fed800a98f0 2024-03-25T18:24:33.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.444+0000 7fed75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:33.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.444+0000 7fed76ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:33.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.637+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fed7c003bb0 con 0x7fed800a98f0 2024-03-25T18:24:33.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.637+0000 7fed75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.637+0000 7fed75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:33.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.637+0000 7fed75ffb640 10 monclient: dump: 2024-03-25T18:24:33.638 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.637+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2218276318 0 0) 0x7fed7c019b10 con 0x7fed800a98f0 2024-03-25T18:24:33.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.638+0000 7fed84b27640 10 monclient: get_auth_request con 0x7fed6406db00 auth_method 0 2024-03-25T18:24:33.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.638+0000 7fed75ffb640 10 client.4173.objecter ms_handle_connect 0x7fed6406db00 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.645+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fed7c0537e0 con 0x7fed800a98f0 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.645+0000 7fed75ffb640 10 client.4173.objecter ms_dispatch 0x7fed80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.645+0000 7fed75ffb640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.645+0000 7fed75ffb640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.646+0000 7fed75ffb640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.646+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed7c04f120 con 0x7fed800a98f0 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.646+0000 7fed75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:33.646+0000 7fed75ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:34.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.004+0000 7fed8758c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.004+0000 7fed8758c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.004+0000 7fed8758c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:34.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.004+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed800627b0 con 0x7fed800a98f0 2024-03-25T18:24:34.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.081+0000 7fed75ffb640 1 -- 172.21.15.176:0/2109939978 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fed7c051220 con 0x7fed800a98f0 2024-03-25T18:24:34.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.081+0000 7fed75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.081+0000 7fed75ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:34.083 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:24:34.083 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 10 librados: watch_flush enter 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 10 librados: watch_flush exit 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 >> v1:172.21.15.176:6800/2598213983 conn(0x7fed6406db00 legacy=0x7fed6406ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:34.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 10 monclient: shutdown 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 >> v1:172.21.15.176:6789/0 conn(0x7fed800a98f0 legacy=0x7fed801dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 shutdown_connections 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.084+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 >> 172.21.15.176:0/2109939978 conn(0x7fed800b7ea0 msgr2=0x7fed80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.085+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 shutdown_connections 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.085+0000 7fed8758c640 1 -- 172.21.15.176:0/2109939978 wait complete. 2024-03-25T18:24:34.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:34.085+0000 7fed8758c640 1 librados: shutdown 2024-03-25T18:24:34.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:37.955 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:06.833 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.018 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:06.896 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.038 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:06.918 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.098 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:06.978 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.164 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:07.058 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.188 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:07.083 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.224 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:07.118 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.226 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:07.120 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.248 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:07.143 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.286 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:07.180 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.454 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:07.356 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.520 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:07.423 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.562 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-03-25T18:24:38.555+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:38.634 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:07.518 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-03-25T18:24:38.642+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:38.693 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:07.577 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.723 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:07.611 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.783 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:07.671 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:38.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-03-25T18:24:38.811+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:38.825 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-03-25T18:24:38.818+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:38.831 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-03-25T18:24:38.824+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:39.015 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:07.892 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.090 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-03-25T18:24:39.082+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:39.142 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.022 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.270 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-03-25T18:24:39.262+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:39.274 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:08.169 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.279 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:08.173 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.298 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:08.192 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.371 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-03-25T18:24:39.364+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:39.377 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:08.255 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.511 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.391 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.559 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:08.462 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.639 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:08.534 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.648 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:08.542 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.662 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:08.556 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.732 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:08.610 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.767 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:08.651 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.825 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.705 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.844 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:08.732 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.941 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:08.843 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:39.999 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:08.894 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.016 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:08.910 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.021 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:08.899 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.022 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:08.916 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.090 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.970 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.142 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:09.025 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.212 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:09.100 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.288 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.183 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.297 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:09.199 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.305 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.200 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.314 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:09.209 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.317 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.195 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.361 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.241 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.495 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:09.378 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.533 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:09.421 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.577 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.472 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.588 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:09.490 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.606 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:09.501 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.612 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.506 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.620 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.497 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.634 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.514 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.783 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:09.666 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.803 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:09.691 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.869 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.764 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.875 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:09.778 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.900 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:09.795 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.903 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.783 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.904 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.798 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.905 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:40.914 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.791 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.967 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:09.861 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.993+0000 7ff28669d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.993+0000 7ff28669d640 10 monclient: build_initial_monmap 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.993+0000 7ff28669d640 10 monclient: monmap: 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:40.995 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:24:40.995597+0000 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:24:40.995597+0000 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:40.996 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.994+0000 7ff28669d640 1 Processor -- start 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.994+0000 7ff28669d640 1 -- start start 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.994+0000 7ff28669d640 10 monclient: init 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-b con 0x7ff2800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:40.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-a con 0x7ff28013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2801430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280142520 con 0x7ff28013f710 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800b28a0 con 0x7ff2801430c0 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800b2a30 con 0x7ff2800a98f0 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff28669d640 10 monclient(hunting): authenticate will time out at 8159.307999s 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff284c3a640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff2801430c0 0x7ff28014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:36060/0 (socket says 172.21.15.176:36060) 2024-03-25T18:24:40.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.995+0000 7ff284c3a640 1 -- 172.21.15.176:0/3365556223 learned_addr learned my addr 172.21.15.176:0/3365556223 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:40.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981611290 0 0) 0x7ff2800b28a0 con 0x7ff2801430c0 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff268003520 con 0x7ff2801430c0 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704293149 0 0) 0x7ff280142520 con 0x7ff28013f710 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.997+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2800b28a0 con 0x7ff28013f710 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386680145 0 0) 0x7ff2800b2a30 con 0x7ff2800a98f0 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:24:41.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:40.998+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280142520 con 0x7ff2800a98f0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.000+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923353781 0 0) 0x7ff268003520 con 0x7ff2801430c0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 >> v1:172.21.15.176:6790/0 conn(0x7ff2800a98f0 legacy=0x7ff2800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 >> v1:172.21.15.176:6789/0 conn(0x7ff28013f710 legacy=0x7ff2801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 1 monclient: found mon.noname-c 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800b44c0 con 0x7ff2801430c0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff278001a10 con 0x7ff2801430c0 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff27effd640 10 monclient: dump: 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff28669d640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:24:41.004 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.001+0000 7ff28669d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:41.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff278001b70 con 0x7ff2801430c0 2024-03-25T18:24:41.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:41.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 1 -- 172.21.15.176:0/3365556223 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff278001f80 con 0x7ff2801430c0 2024-03-25T18:24:41.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff28669d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:41.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff28669d640 4 set_mon_vals no callback set 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.002+0000 7ff27effd640 10 monclient: dump: 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.008 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 10 monclient: shutdown 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 1 -- 172.21.15.176:0/3365556223 >> v1:172.21.15.186:6789/0 conn(0x7ff2801430c0 legacy=0x7ff28014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 1 -- 172.21.15.176:0/3365556223 shutdown_connections 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 1 -- 172.21.15.176:0/3365556223 >> 172.21.15.176:0/3365556223 conn(0x7ff2800b7ea0 msgr2=0x7ff2800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 1 -- 172.21.15.176:0/3365556223 shutdown_connections 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 1 -- 172.21.15.176:0/3365556223 wait complete. 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 10 monclient: build_initial_monmap 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.003+0000 7ff28669d640 10 monclient: monmap: 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:24:41.009 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 librados: starting msgr at 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 librados: starting objecter 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 Processor -- start 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 -- start start 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 librados: setting wanted keys 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 1 librados: calling monclient init 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 10 monclient: init 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:41.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.004+0000 7ff28669d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-b con 0x7ff2800a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-c con 0x7ff28013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 10 monclient(hunting): picked mon.noname-a con 0x7ff2801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280137c60 con 0x7ff2801430c0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280139160 con 0x7ff2800a98f0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2801380f0 con 0x7ff28013f710 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff28669d640 10 monclient(hunting): authenticate will time out at 8159.317916s 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27f7fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7ff28013f710 0x7ff280133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:55846/0 (socket says 172.21.15.176:55846) 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff284c3a640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff2801430c0 0x7ff2801e2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:56700/0 (socket says 172.21.15.176:56700) 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27f7fe640 1 -- 172.21.15.176:0/4249007146 learned_addr learned my addr 172.21.15.176:0/4249007146 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff2801430c0 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff28013f710 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.005+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.006+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff2800a98f0 2024-03-25T18:24:41.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.006+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004811017 0 0) 0x7ff280139160 con 0x7ff2800a98f0 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c004eb0 con 0x7ff2800a98f0 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872250636 0 0) 0x7ff2801380f0 con 0x7ff28013f710 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.007+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280139160 con 0x7ff28013f710 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20157671 0 0) 0x7ff280137c60 con 0x7ff2801430c0 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.008+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2801380f0 con 0x7ff2801430c0 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136180004 0 0) 0x7ff26c004eb0 con 0x7ff2800a98f0 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff280137c60 con 0x7ff2800a98f0 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff270001870 con 0x7ff2800a98f0 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:41.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.011+0000 7ff27cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 69928036 0 0) 0x7ff280139160 con 0x7ff28013f710 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c004eb0 con 0x7ff28013f710 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff2740019b0 con 0x7ff28013f710 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.017 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299176710 0 0) 0x7ff2801380f0 con 0x7ff2801430c0 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff280139160 con 0x7ff2801430c0 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff278001e80 con 0x7ff2801430c0 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.012+0000 7ff27cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.018 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.014+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2019056865 0 0) 0x7ff280137c60 con 0x7ff2800a98f0 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.014+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 >> v1:172.21.15.176:6790/0 conn(0x7ff28013f710 legacy=0x7ff280133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 >> v1:172.21.15.176:6789/0 conn(0x7ff2801430c0 legacy=0x7ff2801e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 1 monclient: found mon.b 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2801397f0 con 0x7ff2800a98f0 2024-03-25T18:24:41.019 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff27cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 10 monclient: _renew_subs 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2801db280 con 0x7ff2800a98f0 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 10 monclient: _renew_subs 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff280133960 con 0x7ff2800a98f0 2024-03-25T18:24:41.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.015+0000 7ff28669d640 1 librados: init done 2024-03-25T18:24:41.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.016+0000 7ff28669d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.016+0000 7ff28669d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.016+0000 7ff28669d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.016+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280139a40 con 0x7ff2800a98f0 2024-03-25T18:24:41.048 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:09.932 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.069 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:09.957 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.156 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.051 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.162 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:10.065 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.176 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.056 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.207 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.101 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.216 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.094 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff270003610 con 0x7ff2800a98f0 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff270001d80 con 0x7ff2800a98f0 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27cff9640 10 monclient: dump: 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:41.261 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:41.262 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:41.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.259+0000 7ff27e7fc640 4 set_mon_vals no callback set 2024-03-25T18:24:41.262 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:10.156 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.260+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7ff270019810 con 0x7ff2800a98f0 2024-03-25T18:24:41.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.261+0000 7ff27f7fe640 10 monclient: get_auth_request con 0x7ff26c06e0f0 auth_method 0 2024-03-25T18:24:41.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.261+0000 7ff27cff9640 10 client.4207.objecter ms_handle_connect 0x7ff26c06e0f0 2024-03-25T18:24:41.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.263+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7ff270053550 con 0x7ff2800a98f0 2024-03-25T18:24:41.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.263+0000 7ff27cff9640 10 client.4207.objecter ms_dispatch 0x7ff280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:41.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.263+0000 7ff27cff9640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:41.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.263+0000 7ff27cff9640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:41.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.263+0000 7ff27cff9640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T18:24:41.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.273+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff27004ee90 con 0x7ff2800a98f0 2024-03-25T18:24:41.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.273+0000 7ff27cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.273+0000 7ff27cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:41.335 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:10.223 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.340 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:10.224 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.448 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.328 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.450 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:10.353 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.452 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.347 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.459+0000 7ff28669d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:41.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.459+0000 7ff28669d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:41.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.459+0000 7ff28669d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:24:41.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.459+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2800627b0 con 0x7ff2800a98f0 2024-03-25T18:24:41.504 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.398 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.504+0000 7ff27cff9640 1 -- 172.21.15.176:0/4249007146 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7ff270050f90 con 0x7ff2800a98f0 2024-03-25T18:24:41.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.504+0000 7ff27cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:41.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.504+0000 7ff27cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:41.505 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:24:41.505 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:41.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 10 librados: watch_flush enter 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 10 librados: watch_flush exit 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 10 client.4207.objecter successfully canceled tick 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 >> v1:172.21.15.176:6800/2598213983 conn(0x7ff26c06e0f0 legacy=0x7ff26c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 10 monclient: shutdown 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 >> v1:172.21.15.186:6789/0 conn(0x7ff2800a98f0 legacy=0x7ff2801dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 shutdown_connections 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 >> 172.21.15.176:0/4249007146 conn(0x7ff2800b7ea0 msgr2=0x7ff28013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 shutdown_connections 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 -- 172.21.15.176:0/4249007146 wait complete. 2024-03-25T18:24:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:41.505+0000 7ff28669d640 1 librados: shutdown 2024-03-25T18:24:41.515 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.393 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:41.558 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:10.453 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.601 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:10.489 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.637 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:10.520 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.725 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.620 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.737 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:10.640 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.798 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.692 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.875 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:10.763 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.925 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.805 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.934 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:10.818 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:41.980 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.860 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.026 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:10.929 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.046 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.924 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.077 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:10.971 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.101 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.979 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.130 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:11.024 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.151 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:11.039 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.229 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:11.113 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.264 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:11.159 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.318 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:11.213 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.319 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:11.213 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.374 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:11.268 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.564 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:11.466 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.624 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:11.526 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.624 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:11.513 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.679 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:11.567 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.766 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:11.649 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:42.820 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:11.704 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:47.050 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:15.930 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:47.554 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:16.448 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:47.653 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:16.531 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:47.661 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:16.556 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:47.960 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:16.854 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:48.095 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:16.997 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:48.146 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:17.034 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:48.322 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:48.408 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:17.292 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.435+0000 7f48de944640 10 monclient: get_monmap_and_config 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.435+0000 7f48de944640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.435+0000 7f48de944640 10 monclient: monmap: 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:24:48.437330+0000 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:24:48.437330+0000 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:48.437 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:48.438 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:48.438 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.436+0000 7f48de944640 1 Processor -- start 2024-03-25T18:24:48.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.436+0000 7f48de944640 1 -- start start 2024-03-25T18:24:48.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.436+0000 7f48de944640 10 monclient: init 2024-03-25T18:24:48.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient(hunting): picked mon.noname-c con 0x7f48d80b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient(hunting): picked mon.noname-b con 0x7f48d80b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient(hunting): picked mon.noname-a con 0x7f48d814be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80af940 con 0x7f48d814be90 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80b0a70 con 0x7f48d80b4d60 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80b0d50 con 0x7f48d80b1730 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.437+0000 7f48de944640 10 monclient(hunting): authenticate will time out at 8166.750630s 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.438+0000 7f48dcee1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f48d814be90 0x7f48d814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:53992/0 (socket says 172.21.15.176:53992) 2024-03-25T18:24:48.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.438+0000 7f48dcee1640 1 -- 172.21.15.176:0/2947716462 learned_addr learned my addr 172.21.15.176:0/2947716462 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637811140 0 0) 0x7f48d80b0d50 con 0x7f48d80b1730 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:24:48.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0003520 con 0x7f48d80b1730 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431705359 0 0) 0x7f48d80af940 con 0x7f48d814be90 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:24:48.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.441+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c0003840 con 0x7f48d814be90 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860655141 0 0) 0x7f48d80b0a70 con 0x7f48d80b4d60 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:24:48.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.442+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d80af940 con 0x7f48d80b4d60 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847267083 0 0) 0x7f48c0003520 con 0x7f48d80b1730 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 >> v1:172.21.15.186:6789/0 conn(0x7f48d80b4d60 legacy=0x7f48d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 >> v1:172.21.15.176:6789/0 conn(0x7f48d814be90 legacy=0x7f48d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80b1030 con 0x7f48d80b1730 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48de944640 5 monclient: authenticate success, global_id 4220 2024-03-25T18:24:48.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.446+0000 7f48de944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f48cc001630 con 0x7f48d80b1730 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: dump: 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:48.450 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48cc0018c0 con 0x7f48d80b1730 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 1 -- 172.21.15.176:0/2947716462 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f48cc001630 con 0x7f48d80b1730 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48de944640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48de944640 4 set_mon_vals no callback set 2024-03-25T18:24:48.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.448+0000 7f48d6ffd640 10 monclient: dump: 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:48.452 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 10 monclient: shutdown 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 1 -- 172.21.15.176:0/2947716462 >> v1:172.21.15.176:6790/0 conn(0x7f48d80b1730 legacy=0x7f48d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 1 -- 172.21.15.176:0/2947716462 shutdown_connections 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 1 -- 172.21.15.176:0/2947716462 >> 172.21.15.176:0/2947716462 conn(0x7f48d80617a0 msgr2=0x7f48d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 1 -- 172.21.15.176:0/2947716462 shutdown_connections 2024-03-25T18:24:48.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.449+0000 7f48de944640 1 -- 172.21.15.176:0/2947716462 wait complete. 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.450+0000 7f48de944640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.450+0000 7f48de944640 10 monclient: monmap: 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 librados: starting msgr at 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 librados: starting objecter 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 Processor -- start 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 -- start start 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 librados: setting wanted keys 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 1 librados: calling monclient init 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 10 monclient: init 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.451+0000 7f48de944640 10 monclient(hunting): picked mon.noname-c con 0x7f48d80b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 10 monclient(hunting): picked mon.noname-a con 0x7f48d80b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 10 monclient(hunting): picked mon.noname-b con 0x7f48d814be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d8137870 con 0x7f48d80b4d60 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d8138180 con 0x7f48d814be90 2024-03-25T18:24:48.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d8138460 con 0x7f48d80b1730 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48de944640 10 monclient(hunting): authenticate will time out at 8166.764985s 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48dcee1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f48d814be90 0x7f48d8144230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:49574/0 (socket says 172.21.15.176:49574) 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48dcee1640 1 -- 172.21.15.176:0/3120423025 learned_addr learned my addr 172.21.15.176:0/3120423025 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48d4ff9640 10 client.?.objecter ms_handle_connect 0x7f48d814be90 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.452+0000 7f48d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.453+0000 7f48d4ff9640 10 client.?.objecter ms_handle_connect 0x7f48d80b4d60 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.453+0000 7f48d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.453+0000 7f48d4ff9640 10 client.?.objecter ms_handle_connect 0x7f48d80b1730 2024-03-25T18:24:48.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.453+0000 7f48d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300646835 0 0) 0x7f48d8137870 con 0x7f48d80b4d60 2024-03-25T18:24:48.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.455+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c4004eb0 con 0x7f48d80b4d60 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399422758 0 0) 0x7f48d8138180 con 0x7f48d814be90 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d8137870 con 0x7f48d814be90 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647867711 0 0) 0x7f48d8138460 con 0x7f48d80b1730 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:24:48.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.456+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d8138180 con 0x7f48d80b1730 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636653993 0 0) 0x7f48c4004eb0 con 0x7f48d80b4d60 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48d8138460 con 0x7f48d80b4d60 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f48cc001d70 con 0x7f48d80b4d60 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.463 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.464 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:48.464 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:48.464 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:48.464 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.460+0000 7f48d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010544666 0 0) 0x7f48d8137870 con 0x7f48d814be90 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48c4004eb0 con 0x7f48d814be90 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f48d0001bb0 con 0x7f48d814be90 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.468 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 131493275 0 0) 0x7f48d8138460 con 0x7f48d80b4d60 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 >> v1:172.21.15.176:6790/0 conn(0x7f48d80b1730 legacy=0x7f48d8137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 >> v1:172.21.15.186:6789/0 conn(0x7f48d814be90 legacy=0x7f48d8144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 monclient: found mon.a 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:48.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8138740 con 0x7f48d80b4d60 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48de944640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48de944640 10 monclient: _renew_subs 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48de944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.463+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d8136bd0 con 0x7f48d80b4d60 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.464+0000 7f48de944640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.464+0000 7f48de944640 10 monclient: _renew_subs 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.464+0000 7f48de944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.464+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d806dea0 con 0x7f48d80b4d60 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.464+0000 7f48de944640 1 librados: init done 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.465+0000 7f48de944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.465+0000 7f48de944640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.465+0000 7f48de944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:48.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.465+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d8140580 con 0x7f48d80b4d60 2024-03-25T18:24:48.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48cc0011f0 con 0x7f48d80b4d60 2024-03-25T18:24:48.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f48cc0038a0 con 0x7f48d80b4d60 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d4ff9640 10 monclient: dump: 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:48.484 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:48.485 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:48.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.480+0000 7f48d67fc640 4 set_mon_vals no callback set 2024-03-25T18:24:48.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.481+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f48cc019930 con 0x7f48d80b4d60 2024-03-25T18:24:48.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.482+0000 7f48d7fff640 10 monclient: get_auth_request con 0x7f48c406dec0 auth_method 0 2024-03-25T18:24:48.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.482+0000 7f48d4ff9640 10 client.4209.objecter ms_handle_connect 0x7f48c406dec0 2024-03-25T18:24:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.483+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f48cc0535f0 con 0x7f48d80b4d60 2024-03-25T18:24:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.483+0000 7f48d4ff9640 10 client.4209.objecter ms_dispatch 0x7f48d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.483+0000 7f48d4ff9640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.483+0000 7f48d4ff9640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.483+0000 7f48d4ff9640 20 client.4209.objecter dump_active .. 0 homeless 2024-03-25T18:24:48.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.816+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48cc052cc0 con 0x7f48d80b4d60 2024-03-25T18:24:48.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.816+0000 7f48d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:48.816+0000 7f48d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:49.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.036+0000 7f48de944640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.036+0000 7f48de944640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.036+0000 7f48de944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:49.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.036+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48d813fe00 con 0x7f48d80b4d60 2024-03-25T18:24:49.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.051+0000 7f48d4ff9640 1 -- 172.21.15.176:0/3120423025 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f48cc04f100 con 0x7f48d80b4d60 2024-03-25T18:24:49.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.051+0000 7f48d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.051+0000 7f48d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:49.054 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:24:49.054 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:49.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 10 librados: watch_flush enter 2024-03-25T18:24:49.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 10 librados: watch_flush exit 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 20 client.4209.objecter shutdown clearing up homeless session... 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 10 client.4209.objecter successfully canceled tick 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 >> v1:172.21.15.176:6800/2598213983 conn(0x7f48c406dec0 legacy=0x7f48c4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 10 monclient: shutdown 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 >> v1:172.21.15.176:6789/0 conn(0x7f48d80b4d60 legacy=0x7f48d8143b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 shutdown_connections 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 >> 172.21.15.176:0/3120423025 conn(0x7f48d80617a0 msgr2=0x7f48d813c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 shutdown_connections 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.054+0000 7f48de944640 1 -- 172.21.15.176:0/3120423025 wait complete. 2024-03-25T18:24:49.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:49.055+0000 7f48de944640 1 librados: shutdown 2024-03-25T18:24:49.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:52.409 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:21.289 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:52.415 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:21.295 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.298 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:22.193 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.302 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:22.197 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.308 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:22.202 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.312 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:22.206 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.536 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:22.430 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.540 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:22.434 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.579 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:22.457 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.583 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:22.461 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.909 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:22.811 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.913 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:22.815 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.924 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:22.812 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:53.928 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:22.816 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:54.165 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:23.049 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:54.171 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:23.055 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.876 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:55.965 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.963+0000 7f54e6f9f640 10 monclient: get_monmap_and_config 2024-03-25T18:24:55.965 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.963+0000 7f54e6f9f640 10 monclient: build_initial_monmap 2024-03-25T18:24:55.965 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.963+0000 7f54e6f9f640 10 monclient: monmap: 2024-03-25T18:24:55.965 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:24:55.965306+0000 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:24:55.965306+0000 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 1 Processor -- start 2024-03-25T18:24:55.967 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 1 -- start start 2024-03-25T18:24:55.967 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 10 monclient: init 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f54e00b49a0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.964+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f54e00a9680 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f54e00b3fa0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e013fe70 con 0x7f54e00b3fa0 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e00ab250 con 0x7f54e00a9680 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e00a9b10 con 0x7f54e00b49a0 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e6f9f640 10 monclient(hunting): authenticate will time out at 8174.277991s 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e579c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f54e00a9680 0x7f54e0145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:51024/0 (socket says 172.21.15.176:51024) 2024-03-25T18:24:55.968 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.965+0000 7f54e579c640 1 -- 172.21.15.176:0/45504175 learned_addr learned my addr 172.21.15.176:0/45504175 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974318078 0 0) 0x7f54e00ab250 con 0x7f54e00a9680 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:24:55.970 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.968+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc003520 con 0x7f54e00a9680 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691472702 0 0) 0x7f54e00a9b10 con 0x7f54e00b49a0 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:24:55.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.969+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e00ab250 con 0x7f54e00b49a0 2024-03-25T18:24:55.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84918954 0 0) 0x7f54cc003520 con 0x7f54e00a9680 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 >> v1:172.21.15.176:6790/0 conn(0x7f54e00b49a0 legacy=0x7f54e013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 >> v1:172.21.15.176:6789/0 conn(0x7f54e00b3fa0 legacy=0x7f54e0145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e0151b50 con 0x7f54e00a9680 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f54d0001690 con 0x7f54e00a9680 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:55.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e4f9b640 10 monclient: dump: 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e6f9f640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:24:55.977 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.973+0000 7f54e6f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54d00017f0 con 0x7f54e00a9680 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 1 -- 172.21.15.176:0/45504175 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f54d0001bc0 con 0x7f54e00a9680 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e4f9b640 10 monclient: dump: 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:55.978 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:55.979 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:55.979 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:55.979 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e6f9f640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:55.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.976+0000 7f54e6f9f640 4 set_mon_vals no callback set 2024-03-25T18:24:55.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 10 monclient: shutdown 2024-03-25T18:24:55.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:55.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 1 -- 172.21.15.176:0/45504175 >> v1:172.21.15.186:6789/0 conn(0x7f54e00a9680 legacy=0x7f54e0145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 1 -- 172.21.15.176:0/45504175 shutdown_connections 2024-03-25T18:24:55.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 1 -- 172.21.15.176:0/45504175 >> 172.21.15.176:0/45504175 conn(0x7f54e00615c0 msgr2=0x7f54e00619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 1 -- 172.21.15.176:0/45504175 shutdown_connections 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.977+0000 7f54e6f9f640 1 -- 172.21.15.176:0/45504175 wait complete. 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 10 monclient: build_initial_monmap 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 10 monclient: monmap: 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:24:55.981 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 librados: starting msgr at 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 librados: starting objecter 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 Processor -- start 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 -- start start 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 librados: setting wanted keys 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 1 librados: calling monclient init 2024-03-25T18:24:55.982 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.978+0000 7f54e6f9f640 10 monclient: init 2024-03-25T18:24:55.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:55.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:55.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f54e00b3fa0 addr v1:172.21.15.176:6790/0 2024-03-25T18:24:55.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f54e00b49a0 addr v1:172.21.15.176:6789/0 2024-03-25T18:24:55.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f54e01db7a0 addr v1:172.21.15.186:6789/0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e013c340 con 0x7f54e00b49a0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e013c5b0 con 0x7f54e01db7a0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e013c9d0 con 0x7f54e00b3fa0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.979+0000 7f54e6f9f640 10 monclient(hunting): authenticate will time out at 8174.292365s 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54e579c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f54e00b49a0 0x7f54e01db090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59628/0 (socket says 172.21.15.176:59628) 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54e579c640 1 -- 172.21.15.176:0/2303611419 learned_addr learned my addr 172.21.15.176:0/2303611419 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54d67fc640 10 client.?.objecter ms_handle_connect 0x7f54e00b49a0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54d67fc640 10 client.?.objecter ms_handle_connect 0x7f54e01db7a0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.980+0000 7f54d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.981+0000 7f54d67fc640 10 client.?.objecter ms_handle_connect 0x7f54e00b3fa0 2024-03-25T18:24:55.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.981+0000 7f54d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510782099 0 0) 0x7f54e013c340 con 0x7f54e00b49a0 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.982+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c4004eb0 con 0x7f54e00b49a0 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640239894 0 0) 0x7f54e013c5b0 con 0x7f54e01db7a0 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:55.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.983+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e013c340 con 0x7f54e01db7a0 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252820134 0 0) 0x7f54e013c9d0 con 0x7f54e00b3fa0 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:24:55.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.984+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e013c5b0 con 0x7f54e00b3fa0 2024-03-25T18:24:55.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.987+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008303315 0 0) 0x7f54c4004eb0 con 0x7f54e00b49a0 2024-03-25T18:24:55.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.987+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54e013c9d0 con 0x7f54e00b49a0 2024-03-25T18:24:55.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.987+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f54d0001bc0 con 0x7f54e00b49a0 2024-03-25T18:24:55.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.987+0000 7f54d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.987+0000 7f54d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.988+0000 7f54d67fc640 10 monclient(hunting): dump: 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.988+0000 7f54d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2017756865 0 0) 0x7f54e013c340 con 0x7f54e01db7a0 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c4004eb0 con 0x7f54e01db7a0 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f54d8001b10 con 0x7f54e01db7a0 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.990+0000 7f54d67fc640 10 monclient(hunting): dump: 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:55.992 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:55.993 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:55.993 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:55.993 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2712121107 0 0) 0x7f54e013c9d0 con 0x7f54e00b49a0 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 >> v1:172.21.15.176:6790/0 conn(0x7f54e00b3fa0 legacy=0x7f54e013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 >> v1:172.21.15.186:6789/0 conn(0x7f54e01db7a0 legacy=0x7f54e01e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 1 monclient: found mon.a 2024-03-25T18:24:55.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e013cc50 con 0x7f54e00b49a0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 10 monclient: _renew_subs 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e013b160 con 0x7f54e00b49a0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 10 monclient: _renew_subs 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e0139f70 con 0x7f54e00b49a0 2024-03-25T18:24:55.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.991+0000 7f54e6f9f640 1 librados: init done 2024-03-25T18:24:55.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.992+0000 7f54e6f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:55.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.992+0000 7f54e6f9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:55.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.992+0000 7f54e6f9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:55.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:55.993+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e00627b0 con 0x7f54e00b49a0 2024-03-25T18:24:56.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54d0004a90 con 0x7f54e00b49a0 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f54d0004e20 con 0x7f54e00b49a0 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d67fc640 10 monclient: dump: 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:24:56.255 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:24:56.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.252+0000 7f54d7fff640 4 set_mon_vals no callback set 2024-03-25T18:24:56.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.253+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f54d0005040 con 0x7f54e00b49a0 2024-03-25T18:24:56.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.254+0000 7f54e5f9d640 10 monclient: get_auth_request con 0x7f54c406df70 auth_method 0 2024-03-25T18:24:56.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.254+0000 7f54d67fc640 10 client.4212.objecter ms_handle_connect 0x7f54c406df70 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f54d0054960 con 0x7f54e00b49a0 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 10 client.4212.objecter ms_dispatch 0x7f54e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54d0050470 con 0x7f54e00b49a0 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:56.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.262+0000 7f54d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:56.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.521+0000 7f54e6f9f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:56.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.521+0000 7f54e6f9f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:56.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.521+0000 7f54e6f9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:24:56.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.521+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54e014e900 con 0x7f54e00b49a0 2024-03-25T18:24:56.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.545+0000 7f54d67fc640 1 -- 172.21.15.176:0/2303611419 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f54d00523a0 con 0x7f54e00b49a0 2024-03-25T18:24:56.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.545+0000 7f54d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:56.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.545+0000 7f54d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:56.547 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:24:56.547 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:56.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 10 librados: watch_flush enter 2024-03-25T18:24:56.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 10 librados: watch_flush exit 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 >> v1:172.21.15.176:6800/2598213983 conn(0x7f54c406df70 legacy=0x7f54c4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 10 monclient: shutdown 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 >> v1:172.21.15.176:6789/0 conn(0x7f54e00b49a0 legacy=0x7f54e01db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 shutdown_connections 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 >> 172.21.15.176:0/2303611419 conn(0x7f54e00615c0 msgr2=0x7f54e00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 shutdown_connections 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 -- 172.21.15.176:0/2303611419 wait complete. 2024-03-25T18:24:56.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:24:56.547+0000 7f54e6f9f640 1 librados: shutdown 2024-03-25T18:24:56.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:56.583 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:25.463 35196== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.513 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:00:26.407 35168== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.565 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:26.459 35195== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.851 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:26.728 35197== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.867 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:26.762 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.076 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:00:26.964 35169== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.565 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:00:27.448 35170== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.659 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:00:27.561 35167== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:03.367 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:03.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.505+0000 7f8c64b32640 10 monclient: get_monmap_and_config 2024-03-25T18:25:03.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.505+0000 7f8c64b32640 10 monclient: build_initial_monmap 2024-03-25T18:25:03.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.505+0000 7f8c64b32640 10 monclient: monmap: 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:03.506424+0000 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:03.506424+0000 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:03.507 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.506+0000 7f8c64b32640 1 Processor -- start 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.506+0000 7f8c64b32640 1 -- start start 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.506+0000 7f8c64b32640 10 monclient: init 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.507+0000 7f8c64b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.507+0000 7f8c64b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.507+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-a con 0x7f8c600a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.507+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-c con 0x7f8c6013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-b con 0x7f8c601430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60142520 con 0x7f8c600a98f0 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600b28a0 con 0x7f8c6013f710 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600b2a30 con 0x7f8c601430c0 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:03.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c64b32640 10 monclient(hunting): authenticate will time out at 8181.820122s 2024-03-25T18:25:03.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c5ed9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f8c601430c0 0x7f8c6014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:38336/0 (socket says 172.21.15.176:38336) 2024-03-25T18:25:03.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.508+0000 7f8c5ed9d640 1 -- 172.21.15.176:0/1737386068 learned_addr learned my addr 172.21.15.176:0/1737386068 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375565167 0 0) 0x7f8c600b2a30 con 0x7f8c601430c0 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 10 monclient(hunting): my global_id is 4232 2024-03-25T18:25:03.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.511+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c003520 con 0x7f8c601430c0 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880899495 0 0) 0x7f8c600b28a0 con 0x7f8c6013f710 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:25:03.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.512+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c600b2a30 con 0x7f8c6013f710 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491011730 0 0) 0x7f8c60142520 con 0x7f8c600a98f0 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c600b28a0 con 0x7f8c600a98f0 2024-03-25T18:25:03.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795354387 0 0) 0x7f8c4c003520 con 0x7f8c601430c0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 >> v1:172.21.15.186:6789/0 conn(0x7f8c6013f710 legacy=0x7f8c601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 >> v1:172.21.15.176:6789/0 conn(0x7f8c600a98f0 legacy=0x7f8c600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600b44c0 con 0x7f8c601430c0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c58001a10 con 0x7f8c601430c0 2024-03-25T18:25:03.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c5dd9b640 10 monclient: dump: 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c64b32640 5 monclient: authenticate success, global_id 4232 2024-03-25T18:25:03.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.515+0000 7f8c64b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.517+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c58001b70 con 0x7f8c601430c0 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.517+0000 7f8c5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.517+0000 7f8c64b32640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.517+0000 7f8c64b32640 4 set_mon_vals no callback set 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c5dd9b640 1 -- 172.21.15.176:0/1737386068 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c58001f80 con 0x7f8c601430c0 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c5dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c5dd9b640 10 monclient: dump: 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.520 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c64b32640 10 monclient: shutdown 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c64b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.518+0000 7f8c64b32640 1 -- 172.21.15.176:0/1737386068 >> v1:172.21.15.176:6790/0 conn(0x7f8c601430c0 legacy=0x7f8c6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.519+0000 7f8c64b32640 1 -- 172.21.15.176:0/1737386068 shutdown_connections 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.519+0000 7f8c64b32640 1 -- 172.21.15.176:0/1737386068 >> 172.21.15.176:0/1737386068 conn(0x7f8c600b7ea0 msgr2=0x7f8c600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.519+0000 7f8c64b32640 1 -- 172.21.15.176:0/1737386068 shutdown_connections 2024-03-25T18:25:03.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.519+0000 7f8c64b32640 1 -- 172.21.15.176:0/1737386068 wait complete. 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.520+0000 7f8c64b32640 10 monclient: build_initial_monmap 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.520+0000 7f8c64b32640 10 monclient: monmap: 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:03.523 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 librados: starting msgr at 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 librados: starting objecter 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 Processor -- start 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 -- start start 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 librados: setting wanted keys 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 1 librados: calling monclient init 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.521+0000 7f8c64b32640 10 monclient: init 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-a con 0x7f8c600a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-b con 0x7f8c6013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient(hunting): picked mon.noname-c con 0x7f8c601430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c601309a0 con 0x7f8c600a98f0 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60135970 con 0x7f8c6013f710 2024-03-25T18:25:03.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60136ca0 con 0x7f8c601430c0 2024-03-25T18:25:03.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:03.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c64b32640 10 monclient(hunting): authenticate will time out at 8181.834257s 2024-03-25T18:25:03.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c57fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8c6013f710 0x7f8c6013b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:51062/0 (socket says 172.21.15.176:51062) 2024-03-25T18:25:03.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.522+0000 7f8c57fff640 1 -- 172.21.15.176:0/2908169763 learned_addr learned my addr 172.21.15.176:0/2908169763 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:03.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c6013f710 2024-03-25T18:25:03.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:03.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c601430c0 2024-03-25T18:25:03.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:03.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter ms_handle_connect 0x7f8c600a98f0 2024-03-25T18:25:03.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.523+0000 7f8c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792189272 0 0) 0x7f8c60135970 con 0x7f8c6013f710 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.525+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c40004eb0 con 0x7f8c6013f710 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253341990 0 0) 0x7f8c60136ca0 con 0x7f8c601430c0 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): my global_id is 4235 2024-03-25T18:25:03.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60135970 con 0x7f8c601430c0 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975042981 0 0) 0x7f8c601309a0 con 0x7f8c600a98f0 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:25:03.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.526+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60136ca0 con 0x7f8c600a98f0 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194685066 0 0) 0x7f8c60135970 con 0x7f8c601430c0 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c601309a0 con 0x7f8c601430c0 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c58001e80 con 0x7f8c601430c0 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.532 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.531+0000 7f8c56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.532+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952255474 0 0) 0x7f8c40004eb0 con 0x7f8c6013f710 2024-03-25T18:25:03.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.532+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c60135970 con 0x7f8c6013f710 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.533+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c48001ab0 con 0x7f8c6013f710 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.533+0000 7f8c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.533+0000 7f8c56ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.533+0000 7f8c56ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.534 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635274612 0 0) 0x7f8c60136ca0 con 0x7f8c600a98f0 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c40004eb0 con 0x7f8c600a98f0 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c50001870 con 0x7f8c600a98f0 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.534+0000 7f8c56ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:03.535 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.536 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377790290 0 0) 0x7f8c601309a0 con 0x7f8c601430c0 2024-03-25T18:25:03.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 >> v1:172.21.15.186:6789/0 conn(0x7f8c6013f710 legacy=0x7f8c6013b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 >> v1:172.21.15.176:6789/0 conn(0x7f8c600a98f0 legacy=0x7f8c6012fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 1 monclient: found mon.c 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60136f80 con 0x7f8c601430c0 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c64b32640 5 monclient: authenticate success, global_id 4235 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c64b32640 10 monclient: _renew_subs 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c64b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:03.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.535+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c60137260 con 0x7f8c601430c0 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.536+0000 7f8c64b32640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.536+0000 7f8c64b32640 10 monclient: _renew_subs 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.536+0000 7f8c64b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.536+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c601363a0 con 0x7f8c601430c0 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.536+0000 7f8c64b32640 1 librados: init done 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.537+0000 7f8c64b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.537+0000 7f8c64b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.537+0000 7f8c64b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:03.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.537+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c600627b0 con 0x7f8c601430c0 2024-03-25T18:25:03.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.748+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c58001fe0 con 0x7f8c601430c0 2024-03-25T18:25:03.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.748+0000 7f8c56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:03.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.748+0000 7f8c5d59a640 4 set_mon_vals no callback set 2024-03-25T18:25:03.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.789+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8c58004fb0 con 0x7f8c601430c0 2024-03-25T18:25:03.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.789+0000 7f8c56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.789+0000 7f8c56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.789+0000 7f8c56ffd640 10 monclient: dump: 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:03.790 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:03.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.792+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f8c5801af80 con 0x7f8c601430c0 2024-03-25T18:25:03.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.793+0000 7f8c5e59c640 10 monclient: get_auth_request con 0x7f8c4006dff0 auth_method 0 2024-03-25T18:25:03.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.793+0000 7f8c56ffd640 10 client.4235.objecter ms_handle_connect 0x7f8c4006dff0 2024-03-25T18:25:03.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.810+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f8c58054ad0 con 0x7f8c601430c0 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.810+0000 7f8c56ffd640 10 client.4235.objecter ms_dispatch 0x7f8c60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.810+0000 7f8c56ffd640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.810+0000 7f8c56ffd640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.810+0000 7f8c56ffd640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.811+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c580505e0 con 0x7f8c601430c0 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.811+0000 7f8c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:03.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:03.811+0000 7f8c56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:04.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.095+0000 7f8c64b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:04.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.095+0000 7f8c64b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:04.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.095+0000 7f8c64b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:04.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.095+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c601373f0 con 0x7f8c601430c0 2024-03-25T18:25:04.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.112+0000 7f8c56ffd640 1 -- 172.21.15.176:0/2908169763 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f8c58052510 con 0x7f8c601430c0 2024-03-25T18:25:04.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.112+0000 7f8c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:04.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.112+0000 7f8c56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:04.113 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:04.113 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:04.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 10 librados: watch_flush enter 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 10 librados: watch_flush exit 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 10 client.4235.objecter successfully canceled tick 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 >> v1:172.21.15.176:6800/2598213983 conn(0x7f8c4006dff0 legacy=0x7f8c400704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 10 monclient: shutdown 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 >> v1:172.21.15.176:6790/0 conn(0x7f8c601430c0 legacy=0x7f8c60130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 shutdown_connections 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.115+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 >> 172.21.15.176:0/2908169763 conn(0x7f8c600b7ea0 msgr2=0x7f8c60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.116+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 shutdown_connections 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.116+0000 7f8c64b32640 1 -- 172.21.15.176:0/2908169763 wait complete. 2024-03-25T18:25:04.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:04.116+0000 7f8c64b32640 1 librados: shutdown 2024-03-25T18:25:04.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:05.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-03-25T18:25:05.277+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:25:06.066 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-03-25T18:25:06.063+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:25:06.199 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-03-25T18:25:06.195+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:25:06.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-03-25T18:25:06.426+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:25:06.532 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-03-25T18:25:06.528+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:25:07.600 INFO:tasks.ceph.osd.0.smithi176.stderr:2024-03-25T18:25:07.595+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:25:07.679 INFO:tasks.ceph.osd.2.smithi176.stderr:2024-03-25T18:25:07.674+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:25:08.362 INFO:tasks.ceph.osd.1.smithi176.stderr:2024-03-25T18:25:08.360+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:25:10.938 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.048+0000 7f123a096640 10 monclient: get_monmap_and_config 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.048+0000 7f123a096640 10 monclient: build_initial_monmap 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.048+0000 7f123a096640 10 monclient: monmap: 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:11.049259+0000 2024-03-25T18:25:11.049 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:11.049259+0000 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:11.050 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.049+0000 7f123a096640 1 Processor -- start 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.049+0000 7f123a096640 1 -- start start 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.049+0000 7f123a096640 10 monclient: init 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:11.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient(hunting): picked mon.noname-b con 0x7f123413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient(hunting): picked mon.noname-c con 0x7f12341430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1234142520 con 0x7f12340a98f0 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340b28a0 con 0x7f12341430c0 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340b2a30 con 0x7f123413f710 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.050+0000 7f123a096640 10 monclient(hunting): authenticate will time out at 8189.362233s 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.051+0000 7f1232ffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f123413f710 0x7f12341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:41250/0 (socket says 172.21.15.176:41250) 2024-03-25T18:25:11.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.051+0000 7f1232ffd640 1 -- 172.21.15.176:0/1966925696 learned_addr learned my addr 172.21.15.176:0/1966925696 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803441886 0 0) 0x7f1234142520 con 0x7f12340a98f0 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 10 monclient(hunting): my global_id is 4245 2024-03-25T18:25:11.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.125+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121c003520 con 0x7f12340a98f0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614396018 0 0) 0x7f121c003520 con 0x7f12340a98f0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 >> v1:172.21.15.176:6790/0 conn(0x7f123413f710 legacy=0x7f12341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 >> v1:172.21.15.186:6789/0 conn(0x7f12341430c0 legacy=0x7f123414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 monclient: found mon.noname-a 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340b44c0 con 0x7f12340a98f0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:11.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1224001690 con 0x7f12340a98f0 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f12327fc640 10 monclient: dump: 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.142 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.143 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f123a096640 5 monclient: authenticate success, global_id 4245 2024-03-25T18:25:11.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.140+0000 7f123a096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12240017f0 con 0x7f12340a98f0 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 1 -- 172.21.15.176:0/1966925696 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1224001c00 con 0x7f12340a98f0 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f123a096640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f123a096640 4 set_mon_vals no callback set 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.157+0000 7f12327fc640 10 monclient: dump: 2024-03-25T18:25:11.158 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.159 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 10 monclient: shutdown 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 1 -- 172.21.15.176:0/1966925696 >> v1:172.21.15.176:6789/0 conn(0x7f12340a98f0 legacy=0x7f12340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 1 -- 172.21.15.176:0/1966925696 shutdown_connections 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 1 -- 172.21.15.176:0/1966925696 >> 172.21.15.176:0/1966925696 conn(0x7f12340b7ea0 msgr2=0x7f12340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 1 -- 172.21.15.176:0/1966925696 shutdown_connections 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.158+0000 7f123a096640 1 -- 172.21.15.176:0/1966925696 wait complete. 2024-03-25T18:25:11.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.159+0000 7f123a096640 10 monclient: build_initial_monmap 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.159+0000 7f123a096640 10 monclient: monmap: 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.159+0000 7f123a096640 1 librados: starting msgr at 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.159+0000 7f123a096640 1 librados: starting objecter 2024-03-25T18:25:11.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.160+0000 7f123a096640 1 Processor -- start 2024-03-25T18:25:11.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.160+0000 7f123a096640 1 -- start start 2024-03-25T18:25:11.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.160+0000 7f123a096640 1 librados: setting wanted keys 2024-03-25T18:25:11.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.160+0000 7f123a096640 1 librados: calling monclient init 2024-03-25T18:25:11.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.160+0000 7f123a096640 10 monclient: init 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient(hunting): picked mon.noname-b con 0x7f12340a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient(hunting): picked mon.noname-c con 0x7f123413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient(hunting): picked mon.noname-a con 0x7f12341430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12341d82d0 con 0x7f12341430c0 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340abb90 con 0x7f12340a98f0 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12341d76a0 con 0x7f123413f710 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:11.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.161+0000 7f123a096640 10 monclient(hunting): authenticate will time out at 8189.473150s 2024-03-25T18:25:11.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.165+0000 7f1233fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f12341430c0 0x7f1234134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54290/0 (socket says 172.21.15.176:54290) 2024-03-25T18:25:11.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.165+0000 7f1233fff640 1 -- 172.21.15.176:0/3706853235 learned_addr learned my addr 172.21.15.176:0/3706853235 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:11.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.167+0000 7f121bfff640 10 client.?.objecter ms_handle_connect 0x7f12341430c0 2024-03-25T18:25:11.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.167+0000 7f121bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012166629 0 0) 0x7f12341d82d0 con 0x7f12341430c0 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:25:11.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.185+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220004eb0 con 0x7f12341430c0 2024-03-25T18:25:11.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.189+0000 7f121bfff640 10 client.?.objecter ms_handle_connect 0x7f12340a98f0 2024-03-25T18:25:11.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.189+0000 7f121bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:11.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.200+0000 7f121bfff640 10 client.?.objecter ms_handle_connect 0x7f123413f710 2024-03-25T18:25:11.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.200+0000 7f121bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930454899 0 0) 0x7f12340abb90 con 0x7f12340a98f0 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:25:11.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.205+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12341d82d0 con 0x7f12340a98f0 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862078172 0 0) 0x7f12341d76a0 con 0x7f123413f710 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:25:11.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.211+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12340abb90 con 0x7f123413f710 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1402305524 0 0) 0x7f1220004eb0 con 0x7f12341430c0 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12341d76a0 con 0x7f12341430c0 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f122c001c30 con 0x7f12341430c0 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.218 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.217+0000 7f121bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681748379 0 0) 0x7f12340abb90 con 0x7f123413f710 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1220004eb0 con 0x7f123413f710 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1228001860 con 0x7f123413f710 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.220+0000 7f121bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.221 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.222 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.222 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.222 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.222 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478558026 0 0) 0x7f12341d82d0 con 0x7f12340a98f0 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12340abb90 con 0x7f12340a98f0 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1224001b80 con 0x7f12340a98f0 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.233+0000 7f121bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.234 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.235 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.235 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.235 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.235 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.235 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729408792 0 0) 0x7f12341d76a0 con 0x7f12341430c0 2024-03-25T18:25:11.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 >> v1:172.21.15.176:6790/0 conn(0x7f123413f710 legacy=0x7f12341d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 >> v1:172.21.15.186:6789/0 conn(0x7f12340a98f0 legacy=0x7f12341385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 1 monclient: found mon.a 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234134720 con 0x7f12341430c0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 10 monclient: _finish_auth 0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f121bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 5 monclient: authenticate success, global_id 4248 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 10 monclient: _renew_subs 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1234134970 con 0x7f12341430c0 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:11.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 10 monclient: _renew_subs 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1234137a10 con 0x7f12341430c0 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.235+0000 7f123a096640 1 librados: init done 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.236+0000 7f123a096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.236+0000 7f123a096640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:11.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.236+0000 7f123a096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.239 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.236+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1234074cc0 con 0x7f12341430c0 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f122c003990 con 0x7f12341430c0 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f122c003da0 con 0x7f12341430c0 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f121bfff640 10 monclient: dump: 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:11.592 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:11.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.591+0000 7f1231ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:11.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.592+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f122c003fc0 con 0x7f12341430c0 2024-03-25T18:25:11.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.592+0000 7f12337fe640 10 monclient: get_auth_request con 0x7f122006e0f0 auth_method 0 2024-03-25T18:25:11.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.592+0000 7f121bfff640 10 client.4248.objecter ms_handle_connect 0x7f122006e0f0 2024-03-25T18:25:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f122c054a40 con 0x7f12341430c0 2024-03-25T18:25:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 10 client.4248.objecter ms_dispatch 0x7f1234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-25T18:25:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f122c050550 con 0x7f12341430c0 2024-03-25T18:25:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.597+0000 7f121bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:11.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.849+0000 7f123a096640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.849+0000 7f123a096640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.849+0000 7f123a096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:11.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.849+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12341d7830 con 0x7f12341430c0 2024-03-25T18:25:11.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.875+0000 7f121bfff640 1 -- 172.21.15.176:0/3706853235 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f122c052480 con 0x7f12341430c0 2024-03-25T18:25:11.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.875+0000 7f121bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:11.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.875+0000 7f121bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:11.876 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:11.877 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 10 librados: watch_flush enter 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 10 librados: watch_flush exit 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 10 client.4248.objecter successfully canceled tick 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 >> v1:172.21.15.176:6800/2598213983 conn(0x7f122006e0f0 legacy=0x7f12200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 10 monclient: shutdown 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 >> v1:172.21.15.176:6789/0 conn(0x7f12341430c0 legacy=0x7f1234134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 shutdown_connections 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 >> 172.21.15.176:0/3706853235 conn(0x7f12340b7ea0 msgr2=0x7f1234074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 shutdown_connections 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 -- 172.21.15.176:0/3706853235 wait complete. 2024-03-25T18:25:11.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:11.878+0000 7f123a096640 1 librados: shutdown 2024-03-25T18:25:11.892 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:18.699 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.797+0000 7f070eba8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.797+0000 7f070eba8640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.797+0000 7f070eba8640 10 monclient: monmap: 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:18.798 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:18.798614+0000 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:18.798614+0000 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:18.799 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:18.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.798+0000 7f070eba8640 1 Processor -- start 2024-03-25T18:25:18.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.798+0000 7f070eba8640 1 -- start start 2024-03-25T18:25:18.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.798+0000 7f070eba8640 10 monclient: init 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-b con 0x7f07080b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-c con 0x7f07080b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-a con 0x7f070814be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080af940 con 0x7f070814be90 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080b0a70 con 0x7f07080b1730 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080b0d50 con 0x7f07080b4d60 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.799+0000 7f070eba8640 10 monclient(hunting): authenticate will time out at 8197.111038s 2024-03-25T18:25:18.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.809+0000 7f070c944640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f07080b4d60 0x7f070813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:36154/0 (socket says 172.21.15.176:36154) 2024-03-25T18:25:18.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.809+0000 7f070c944640 1 -- 172.21.15.176:0/412727944 learned_addr learned my addr 172.21.15.176:0/412727944 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002550546 0 0) 0x7f07080b0a70 con 0x7f07080b1730 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:25:18.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.823+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f8003520 con 0x7f07080b1730 2024-03-25T18:25:18.826 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762684815 0 0) 0x7f07080b0d50 con 0x7f07080b4d60 2024-03-25T18:25:18.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:25:18.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.825+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07080b0a70 con 0x7f07080b4d60 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835458471 0 0) 0x7f06f8003520 con 0x7f07080b1730 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 >> v1:172.21.15.176:6790/0 conn(0x7f07080b4d60 legacy=0x7f070813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 >> v1:172.21.15.176:6789/0 conn(0x7f070814be90 legacy=0x7f070814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080b1030 con 0x7f07080b1730 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f06f4001690 con 0x7f07080b1730 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f07077fe640 10 monclient: dump: 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:18.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f070eba8640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:25:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.832+0000 7f070eba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f40017f0 con 0x7f07080b1730 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 1 -- 172.21.15.176:0/412727944 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f06f4001c00 con 0x7f07080b1730 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f07077fe640 10 monclient: dump: 2024-03-25T18:25:18.847 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f070eba8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:18.848 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.846+0000 7f070eba8640 4 set_mon_vals no callback set 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 10 monclient: shutdown 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 1 -- 172.21.15.176:0/412727944 >> v1:172.21.15.186:6789/0 conn(0x7f07080b1730 legacy=0x7f07080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 1 -- 172.21.15.176:0/412727944 shutdown_connections 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 1 -- 172.21.15.176:0/412727944 >> 172.21.15.176:0/412727944 conn(0x7f07080617a0 msgr2=0x7f0708061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 1 -- 172.21.15.176:0/412727944 shutdown_connections 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.847+0000 7f070eba8640 1 -- 172.21.15.176:0/412727944 wait complete. 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.848+0000 7f070eba8640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.848+0000 7f070eba8640 10 monclient: monmap: 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:18.850 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 librados: starting msgr at 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 librados: starting objecter 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 Processor -- start 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 -- start start 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 librados: setting wanted keys 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 1 librados: calling monclient init 2024-03-25T18:25:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.849+0000 7f070eba8640 10 monclient: init 2024-03-25T18:25:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-c con 0x7f07080b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-a con 0x7f07080b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient(hunting): picked mon.noname-b con 0x7f070814be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07081d83d0 con 0x7f07080b4d60 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07081d77a0 con 0x7f070814be90 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0708138270 con 0x7f07080b1730 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.850+0000 7f070eba8640 10 monclient(hunting): authenticate will time out at 8197.162308s 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.853+0000 7f070d145640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f070814be90 0x7f0708142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:41906/0 (socket says 172.21.15.176:41906) 2024-03-25T18:25:18.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.853+0000 7f070d145640 1 -- 172.21.15.176:0/2136568522 learned_addr learned my addr 172.21.15.176:0/2136568522 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:18.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.856+0000 7f07057fa640 10 client.?.objecter ms_handle_connect 0x7f070814be90 2024-03-25T18:25:18.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.856+0000 7f07057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.862+0000 7f07057fa640 10 client.?.objecter ms_handle_connect 0x7f07080b4d60 2024-03-25T18:25:18.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.862+0000 7f07057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.862+0000 7f07057fa640 10 client.?.objecter ms_handle_connect 0x7f07080b1730 2024-03-25T18:25:18.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.862+0000 7f07057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014256327 0 0) 0x7f07081d77a0 con 0x7f070814be90 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:25:18.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.869+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06ec004eb0 con 0x7f070814be90 2024-03-25T18:25:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500752240 0 0) 0x7f0708138270 con 0x7f07080b1730 2024-03-25T18:25:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:25:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.874+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07081d77a0 con 0x7f07080b1730 2024-03-25T18:25:18.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921497921 0 0) 0x7f07081d83d0 con 0x7f07080b4d60 2024-03-25T18:25:18.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:25:18.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.882+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0708138270 con 0x7f07080b4d60 2024-03-25T18:25:18.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970801011 0 0) 0x7f06ec004eb0 con 0x7f070814be90 2024-03-25T18:25:18.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07081d83d0 con 0x7f070814be90 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0700001c90 con 0x7f070814be90 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 10 monclient(hunting): dump: 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:18.886 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:18.887 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:18.887 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:18.887 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:18.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.884+0000 7f07057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212767131 0 0) 0x7f07081d83d0 con 0x7f070814be90 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 >> v1:172.21.15.176:6790/0 conn(0x7f07080b1730 legacy=0x7f07081d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 >> v1:172.21.15.176:6789/0 conn(0x7f07080b4d60 legacy=0x7f07081405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 1 monclient: found mon.b 2024-03-25T18:25:18.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708140d00 con 0x7f070814be90 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f07057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 10 monclient: _renew_subs 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0708138a80 con 0x7f070814be90 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 10 monclient: _renew_subs 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0708143640 con 0x7f070814be90 2024-03-25T18:25:18.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.890+0000 7f070eba8640 1 librados: init done 2024-03-25T18:25:18.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.891+0000 7f070eba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.891+0000 7f070eba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.891+0000 7f070eba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:18.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:18.891+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0708140ef0 con 0x7f070814be90 2024-03-25T18:25:19.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0700003990 con 0x7f070814be90 2024-03-25T18:25:19.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0700003b40 con 0x7f070814be90 2024-03-25T18:25:19.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:19.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f07057fa640 10 monclient: dump: 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:19.175 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:19.176 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:19.176 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:19.176 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:19.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.172+0000 7f0706ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:19.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.173+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f0700003d60 con 0x7f070814be90 2024-03-25T18:25:19.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.174+0000 7f070c944640 10 monclient: get_auth_request con 0x7f06ec06de20 auth_method 0 2024-03-25T18:25:19.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.174+0000 7f07057fa640 10 client.4261.objecter ms_handle_connect 0x7f06ec06de20 2024-03-25T18:25:19.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f07000548a0 con 0x7f070814be90 2024-03-25T18:25:19.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 10 client.4261.objecter ms_dispatch 0x7f0708000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:19.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:19.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:19.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:25:19.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07000503b0 con 0x7f070814be90 2024-03-25T18:25:19.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.178+0000 7f07057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:19.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.429+0000 7f070eba8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.429+0000 7f070eba8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.429+0000 7f070eba8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:19.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.429+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07080afbd0 con 0x7f070814be90 2024-03-25T18:25:19.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.460+0000 7f07057fa640 1 -- 172.21.15.176:0/2136568522 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f07000522e0 con 0x7f070814be90 2024-03-25T18:25:19.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.460+0000 7f07057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:19.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.460+0000 7f07057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:19.462 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:19.463 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:19.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 10 librados: watch_flush enter 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 10 librados: watch_flush exit 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 >> v1:172.21.15.176:6800/2598213983 conn(0x7f06ec06de20 legacy=0x7f06ec0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 10 monclient: shutdown 2024-03-25T18:25:19.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.463+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 >> v1:172.21.15.186:6789/0 conn(0x7f070814be90 legacy=0x7f0708142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.464+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 shutdown_connections 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.464+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 >> 172.21.15.176:0/2136568522 conn(0x7f07080617a0 msgr2=0x7f0708061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.464+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 shutdown_connections 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.464+0000 7f070eba8640 1 -- 172.21.15.176:0/2136568522 wait complete. 2024-03-25T18:25:19.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:19.464+0000 7f070eba8640 1 librados: shutdown 2024-03-25T18:25:19.476 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:22.358 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:24.114 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.423 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:24.179 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.426 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:24.160 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.471 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:24.227 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.484 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:24.219 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.535 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:24.291 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.656 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:24.412 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.708 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:24.442 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.775 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:24.532 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:26.284 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 10 monclient: get_monmap_and_config 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 10 monclient: build_initial_monmap 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 10 monclient: monmap: 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:26.399910+0000 2024-03-25T18:25:26.400 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:26.399910+0000 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 1 Processor -- start 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 1 -- start start 2024-03-25T18:25:26.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.399+0000 7f5ed02e2640 10 monclient: init 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec806adf0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec8143660 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec8148c60 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80b1180 con 0x7f5ec8143660 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80b13f0 con 0x7f5ec8148c60 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80afde0 con 0x7f5ec806adf0 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:26.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.400+0000 7f5ed02e2640 10 monclient(hunting): authenticate will time out at 8204.712527s 2024-03-25T18:25:26.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.408+0000 7f5ecd87d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f5ec8143660 0x7f5ec8143af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:50410/0 (socket says 172.21.15.176:50410) 2024-03-25T18:25:26.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.408+0000 7f5ecd87d640 1 -- 172.21.15.176:0/2437903581 learned_addr learned my addr 172.21.15.176:0/2437903581 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154718080 0 0) 0x7f5ec80afde0 con 0x7f5ec806adf0 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:25:26.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.417+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb4003520 con 0x7f5ec806adf0 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770089576 0 0) 0x7f5ec80b1180 con 0x7f5ec8143660 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:25:26.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.425+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec80afde0 con 0x7f5ec8143660 2024-03-25T18:25:26.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207621328 0 0) 0x7f5eb4003520 con 0x7f5ec806adf0 2024-03-25T18:25:26.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 >> v1:172.21.15.186:6789/0 conn(0x7f5ec8148c60 legacy=0x7f5ec814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 >> v1:172.21.15.176:6789/0 conn(0x7f5ec8143660 legacy=0x7f5ec8143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: _finish_hunting 0 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 monclient: found mon.noname-b 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec80b0300 con 0x7f5ec806adf0 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: _finish_auth 0 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: _check_auth_tickets 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5ec4001690 con 0x7f5ec806adf0 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ed02e2640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ed02e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ecd07c640 10 monclient: dump: 2024-03-25T18:25:26.431 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.429+0000 7f5ed02e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:26.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec40017f0 con 0x7f5ec806adf0 2024-03-25T18:25:26.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 1 -- 172.21.15.176:0/2437903581 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5ec4001c00 con 0x7f5ec806adf0 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ed02e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ed02e2640 4 set_mon_vals no callback set 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.438+0000 7f5ecd07c640 10 monclient: dump: 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.439 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.440 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 10 monclient: shutdown 2024-03-25T18:25:26.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 1 -- 172.21.15.176:0/2437903581 >> v1:172.21.15.176:6790/0 conn(0x7f5ec806adf0 legacy=0x7f5ec80af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 1 -- 172.21.15.176:0/2437903581 shutdown_connections 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 1 -- 172.21.15.176:0/2437903581 >> 172.21.15.176:0/2437903581 conn(0x7f5ec80b4b80 msgr2=0x7f5ec80b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 1 -- 172.21.15.176:0/2437903581 shutdown_connections 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.439+0000 7f5ed02e2640 1 -- 172.21.15.176:0/2437903581 wait complete. 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 10 monclient: build_initial_monmap 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 10 monclient: monmap: 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:26.441 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:26.442 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:26.442 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 librados: starting msgr at 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 librados: starting objecter 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 Processor -- start 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 -- start start 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 librados: setting wanted keys 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 1 librados: calling monclient init 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.440+0000 7f5ed02e2640 10 monclient: init 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec806adf0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec8143660 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec8148c60 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec81da9f0 con 0x7f5ec8143660 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec81db590 con 0x7f5ec806adf0 2024-03-25T18:25:26.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec81db720 con 0x7f5ec8148c60 2024-03-25T18:25:26.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:26.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.441+0000 7f5ed02e2640 10 monclient(hunting): authenticate will time out at 8204.753533s 2024-03-25T18:25:26.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.449+0000 7f5ece07e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5ec806adf0 0x7f5ec813e150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:50654/0 (socket says 172.21.15.176:50654) 2024-03-25T18:25:26.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.449+0000 7f5ece07e640 1 -- 172.21.15.176:0/3296632906 learned_addr learned my addr 172.21.15.176:0/3296632906 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:26.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.449+0000 7f5ecd87d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f5ec8143660 0x7f5ec81dae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:50436/0 (socket says 172.21.15.176:50436) 2024-03-25T18:25:26.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.450+0000 7f5ebeffd640 10 client.?.objecter ms_handle_connect 0x7f5ec806adf0 2024-03-25T18:25:26.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.450+0000 7f5ebeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.451+0000 7f5ebeffd640 10 client.?.objecter ms_handle_connect 0x7f5ec8143660 2024-03-25T18:25:26.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.451+0000 7f5ebeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.456+0000 7f5ebeffd640 10 client.?.objecter ms_handle_connect 0x7f5ec8148c60 2024-03-25T18:25:26.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.456+0000 7f5ebeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555138201 0 0) 0x7f5ec81db720 con 0x7f5ec8148c60 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac004eb0 con 0x7f5ec8148c60 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487031703 0 0) 0x7f5ec81da9f0 con 0x7f5ec8143660 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:25:26.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.473+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec81db720 con 0x7f5ec8143660 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842356010 0 0) 0x7f5ec81db590 con 0x7f5ec806adf0 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:25:26.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.476+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec81da9f0 con 0x7f5ec806adf0 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246281942 0 0) 0x7f5ec81da9f0 con 0x7f5ec806adf0 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec81db590 con 0x7f5ec806adf0 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5ec4001b00 con 0x7f5ec806adf0 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.491 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.490+0000 7f5ebeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:26.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 994138322 0 0) 0x7f5eac004eb0 con 0x7f5ec8148c60 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec81da9f0 con 0x7f5ec8148c60 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5ec0001b90 con 0x7f5ec8148c60 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.504+0000 7f5ebeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.505 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 679275565 0 0) 0x7f5ec81db720 con 0x7f5ec8143660 2024-03-25T18:25:26.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eac004eb0 con 0x7f5ec8143660 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5eb80018b0 con 0x7f5ec8143660 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.510+0000 7f5ebeffd640 10 monclient(hunting): dump: 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.511 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3748816380 0 0) 0x7f5eac004eb0 con 0x7f5ec8143660 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 >> v1:172.21.15.176:6790/0 conn(0x7f5ec8148c60 legacy=0x7f5ec81e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 >> v1:172.21.15.186:6789/0 conn(0x7f5ec806adf0 legacy=0x7f5ec813e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 1 monclient: found mon.a 2024-03-25T18:25:26.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8133d60 con 0x7f5ec8143660 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 10 monclient: _renew_subs 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec8133fb0 con 0x7f5ec8143660 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 10 monclient: _renew_subs 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:26.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec8134970 con 0x7f5ec8143660 2024-03-25T18:25:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.591+0000 7f5ed02e2640 1 librados: init done 2024-03-25T18:25:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.592+0000 7f5ed02e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.592+0000 7f5ed02e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.592+0000 7f5ed02e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.592+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec80627b0 con 0x7f5ec8143660 2024-03-25T18:25:26.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.600+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb8001d00 con 0x7f5ec8143660 2024-03-25T18:25:26.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.600+0000 7f5ebeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:26.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.600+0000 7f5ecc87b640 4 set_mon_vals no callback set 2024-03-25T18:25:26.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.641+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5eb8004980 con 0x7f5ec8143660 2024-03-25T18:25:26.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.641+0000 7f5ebeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.641+0000 7f5ebeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.641+0000 7f5ebeffd640 10 monclient: dump: 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:26.642 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:26.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.848+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f5eb801a9d0 con 0x7f5ec8143660 2024-03-25T18:25:26.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.849+0000 7f5ece07e640 10 monclient: get_auth_request con 0x7f5eac06e110 auth_method 0 2024-03-25T18:25:26.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.849+0000 7f5ebeffd640 10 client.4260.objecter ms_handle_connect 0x7f5eac06e110 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f5eb8054780 con 0x7f5ec8143660 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 10 client.4260.objecter ms_dispatch 0x7f5ec8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eb80500c0 con 0x7f5ec8143660 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:26.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:26.857+0000 7f5ebeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:27.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.084+0000 7f5ed02e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.084+0000 7f5ed02e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.084+0000 7f5ed02e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:27.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.084+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ec8142b70 con 0x7f5ec8143660 2024-03-25T18:25:27.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.133+0000 7f5ebeffd640 1 -- 172.21.15.176:0/3296632906 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f5eb80521c0 con 0x7f5ec8143660 2024-03-25T18:25:27.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.133+0000 7f5ebeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.133+0000 7f5ebeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:27.135 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:27.135 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 10 librados: watch_flush enter 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 10 librados: watch_flush exit 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 10 client.4260.objecter successfully canceled tick 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 >> v1:172.21.15.176:6800/2598213983 conn(0x7f5eac06e110 legacy=0x7f5eac0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 10 monclient: shutdown 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:27.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 >> v1:172.21.15.176:6789/0 conn(0x7f5ec8143660 legacy=0x7f5ec81dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 shutdown_connections 2024-03-25T18:25:27.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 >> 172.21.15.176:0/3296632906 conn(0x7f5ec80b4b80 msgr2=0x7f5ec813a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:27.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 shutdown_connections 2024-03-25T18:25:27.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 -- 172.21.15.176:0/3296632906 wait complete. 2024-03-25T18:25:27.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:27.136+0000 7f5ed02e2640 1 librados: shutdown 2024-03-25T18:25:27.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:33.957 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.072+0000 7f2427554640 10 monclient: get_monmap_and_config 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.072+0000 7f2427554640 10 monclient: build_initial_monmap 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.072+0000 7f2427554640 10 monclient: monmap: 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:34.073279+0000 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:34.073279+0000 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:34.073 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:34.074 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:34.074 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:34.074 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:34.074 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 1 Processor -- start 2024-03-25T18:25:34.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 1 -- start start 2024-03-25T18:25:34.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient: init 2024-03-25T18:25:34.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:34.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient(hunting): picked mon.noname-b con 0x7f24200a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient(hunting): picked mon.noname-a con 0x7f242013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 10 monclient(hunting): picked mon.noname-c con 0x7f24201430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.073+0000 7f2427554640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420142520 con 0x7f242013f710 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.074+0000 7f2427554640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200b28a0 con 0x7f24201430c0 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.074+0000 7f2427554640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200b2a30 con 0x7f24200a98f0 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.074+0000 7f2427554640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:34.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.074+0000 7f2427554640 10 monclient(hunting): authenticate will time out at 8212.385897s 2024-03-25T18:25:34.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.088+0000 7f2425af1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f24201430c0 0x7f242014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:55376/0 (socket says 172.21.15.176:55376) 2024-03-25T18:25:34.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.088+0000 7f2425af1640 1 -- 172.21.15.176:0/3736639236 learned_addr learned my addr 172.21.15.176:0/3736639236 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:34.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055966236 0 0) 0x7f24200b28a0 con 0x7f24201430c0 2024-03-25T18:25:34.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:25:34.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.093+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c003520 con 0x7f24201430c0 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106863361 0 0) 0x7f24200b2a30 con 0x7f24200a98f0 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:25:34.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.102+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24200b28a0 con 0x7f24200a98f0 2024-03-25T18:25:34.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2671431236 0 0) 0x7f240c003520 con 0x7f24201430c0 2024-03-25T18:25:34.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 >> v1:172.21.15.176:6790/0 conn(0x7f24200a98f0 legacy=0x7f24200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 >> v1:172.21.15.176:6789/0 conn(0x7f242013f710 legacy=0x7f24201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:34.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 monclient: found mon.noname-c 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24200b44c0 con 0x7f24201430c0 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2418001a10 con 0x7f24201430c0 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2427554640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2427554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2417fff640 10 monclient: dump: 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.105 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.103+0000 7f2427554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2418001b70 con 0x7f24201430c0 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 1 -- 172.21.15.176:0/3736639236 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2418001f80 con 0x7f24201430c0 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2427554640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2427554640 4 set_mon_vals no callback set 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.111+0000 7f2417fff640 10 monclient: dump: 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.112 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:34.113 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:34.113 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:34.113 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:34.113 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:34.113 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 10 monclient: shutdown 2024-03-25T18:25:34.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:34.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 1 -- 172.21.15.176:0/3736639236 >> v1:172.21.15.186:6789/0 conn(0x7f24201430c0 legacy=0x7f242014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 1 -- 172.21.15.176:0/3736639236 shutdown_connections 2024-03-25T18:25:34.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 1 -- 172.21.15.176:0/3736639236 >> 172.21.15.176:0/3736639236 conn(0x7f24200b7ea0 msgr2=0x7f24200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 1 -- 172.21.15.176:0/3736639236 shutdown_connections 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.112+0000 7f2427554640 1 -- 172.21.15.176:0/3736639236 wait complete. 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 10 monclient: build_initial_monmap 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 10 monclient: monmap: 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 librados: starting msgr at 2024-03-25T18:25:34.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 librados: starting objecter 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 Processor -- start 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 -- start start 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 librados: setting wanted keys 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 1 librados: calling monclient init 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.113+0000 7f2427554640 10 monclient: init 2024-03-25T18:25:34.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient(hunting): picked mon.noname-c con 0x7f24200a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient(hunting): picked mon.noname-a con 0x7f242013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient(hunting): picked mon.noname-b con 0x7f24201430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420137c60 con 0x7f242013f710 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420139160 con 0x7f24201430c0 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24201380f0 con 0x7f24200a98f0 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:34.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.114+0000 7f2427554640 10 monclient(hunting): authenticate will time out at 8212.426675s 2024-03-25T18:25:34.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.129+0000 7f24252f0640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f24200a98f0 0x7f24201dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:37642/0 (socket says 172.21.15.176:37642) 2024-03-25T18:25:34.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.129+0000 7f24252f0640 1 -- 172.21.15.176:0/571033707 learned_addr learned my addr 172.21.15.176:0/571033707 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:34.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.131+0000 7f2415ffb640 10 client.?.objecter ms_handle_connect 0x7f24200a98f0 2024-03-25T18:25:34.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.131+0000 7f2415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:34.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.131+0000 7f2415ffb640 10 client.?.objecter ms_handle_connect 0x7f242013f710 2024-03-25T18:25:34.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.131+0000 7f2415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:34.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.135+0000 7f2415ffb640 10 client.?.objecter ms_handle_connect 0x7f24201430c0 2024-03-25T18:25:34.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.135+0000 7f2415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228144141 0 0) 0x7f2420139160 con 0x7f24201430c0 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:25:34.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.148+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2404004eb0 con 0x7f24201430c0 2024-03-25T18:25:34.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059573422 0 0) 0x7f2420137c60 con 0x7f242013f710 2024-03-25T18:25:34.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:25:34.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.156+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420139160 con 0x7f242013f710 2024-03-25T18:25:34.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.157+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287215327 0 0) 0x7f24201380f0 con 0x7f24200a98f0 2024-03-25T18:25:34.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.157+0000 7f2415ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.157+0000 7f2415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.157+0000 7f2415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.157+0000 7f2415ffb640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:25:34.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.158+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420137c60 con 0x7f24200a98f0 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491308391 0 0) 0x7f2404004eb0 con 0x7f24201430c0 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24201380f0 con 0x7f24201430c0 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2418001e80 con 0x7f24201430c0 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.165 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.164+0000 7f2415ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2074700946 0 0) 0x7f24201380f0 con 0x7f24201430c0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 >> v1:172.21.15.176:6790/0 conn(0x7f24200a98f0 legacy=0x7f24201dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 >> v1:172.21.15.176:6789/0 conn(0x7f242013f710 legacy=0x7f2420133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 1 monclient: found mon.b 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24201397f0 con 0x7f24201430c0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:34.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2415ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 10 monclient: _renew_subs 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24201db280 con 0x7f24201430c0 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 10 monclient: _renew_subs 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2420133960 con 0x7f24201430c0 2024-03-25T18:25:34.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.170+0000 7f2427554640 1 librados: init done 2024-03-25T18:25:34.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.171+0000 7f2427554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:34.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.171+0000 7f2427554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:34.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.171+0000 7f2427554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.171+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24201db5d0 con 0x7f24201430c0 2024-03-25T18:25:34.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.386+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2418002020 con 0x7f24201430c0 2024-03-25T18:25:34.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.386+0000 7f2415ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:34.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.386+0000 7f24177fe640 4 set_mon_vals no callback set 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.450+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2418004fb0 con 0x7f24201430c0 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.450+0000 7f2415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.450+0000 7f2415ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.450+0000 7f2415ffb640 10 monclient: dump: 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:34.451 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:34.452 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:34.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.451+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f241801af80 con 0x7f24201430c0 2024-03-25T18:25:34.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.452+0000 7f24252f0640 10 monclient: get_auth_request con 0x7f240406ddf0 auth_method 0 2024-03-25T18:25:34.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.452+0000 7f2415ffb640 10 client.4270.objecter ms_handle_connect 0x7f240406ddf0 2024-03-25T18:25:34.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.454+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f2418054b10 con 0x7f24201430c0 2024-03-25T18:25:34.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.454+0000 7f2415ffb640 10 client.4270.objecter ms_dispatch 0x7f2420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:34.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.454+0000 7f2415ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:34.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.454+0000 7f2415ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:34.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.454+0000 7f2415ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:25:34.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.456+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2418050450 con 0x7f24201430c0 2024-03-25T18:25:34.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.456+0000 7f2415ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:34.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.456+0000 7f2415ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:34.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.672+0000 7f2427554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.672+0000 7f2427554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.672+0000 7f2427554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:25:34.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.672+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2420137e10 con 0x7f24201430c0 2024-03-25T18:25:34.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.752+0000 7f2415ffb640 1 -- 172.21.15.176:0/571033707 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f2420137e10 con 0x7f24201430c0 2024-03-25T18:25:34.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.752+0000 7f2415ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:34.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.752+0000 7f2415ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:34.753 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:34.753 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:34.755 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 10 librados: watch_flush enter 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 10 librados: watch_flush exit 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 >> v1:172.21.15.176:6800/2598213983 conn(0x7f240406ddf0 legacy=0x7f24040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 10 monclient: shutdown 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 >> v1:172.21.15.186:6789/0 conn(0x7f24201430c0 legacy=0x7f24201e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 shutdown_connections 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.754+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 >> 172.21.15.176:0/571033707 conn(0x7f24200b7ea0 msgr2=0x7f242013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.755+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 shutdown_connections 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.755+0000 7f2427554640 1 -- 172.21.15.176:0/571033707 wait complete. 2024-03-25T18:25:34.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:34.755+0000 7f2427554640 1 librados: shutdown 2024-03-25T18:25:34.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:41.573 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.686+0000 7f15874af640 10 monclient: get_monmap_and_config 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.686+0000 7f15874af640 10 monclient: build_initial_monmap 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.686+0000 7f15874af640 10 monclient: monmap: 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:41.687156+0000 2024-03-25T18:25:41.687 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:41.687156+0000 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:41.688 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.687+0000 7f15874af640 1 Processor -- start 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.687+0000 7f15874af640 1 -- start start 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.687+0000 7f15874af640 10 monclient: init 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient(hunting): picked mon.noname-c con 0x7f15800a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:41.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient(hunting): picked mon.noname-b con 0x7f158013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient(hunting): picked mon.noname-a con 0x7f15801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580142520 con 0x7f15801430c0 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15800b28a0 con 0x7f15800a98f0 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15800b2a30 con 0x7f158013f710 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:41.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.688+0000 7f15874af640 10 monclient(hunting): authenticate will time out at 8220.000046s 2024-03-25T18:25:41.705 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.705+0000 7f158524b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f15800a98f0 0x7f15800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:35120/0 (socket says 172.21.15.176:35120) 2024-03-25T18:25:41.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.705+0000 7f158524b640 1 -- 172.21.15.176:0/186038042 learned_addr learned my addr 172.21.15.176:0/186038042 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:41.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117691771 0 0) 0x7f15800b28a0 con 0x7f15800a98f0 2024-03-25T18:25:41.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:25:41.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.713+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c003520 con 0x7f15800a98f0 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830910325 0 0) 0x7f1580142520 con 0x7f15801430c0 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:25:41.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.716+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15800b28a0 con 0x7f15801430c0 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252642959 0 0) 0x7f15800b2a30 con 0x7f158013f710 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:25:41.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.720+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580142520 con 0x7f158013f710 2024-03-25T18:25:41.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487954148 0 0) 0x7f156c003520 con 0x7f15800a98f0 2024-03-25T18:25:41.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 >> v1:172.21.15.176:6790/0 conn(0x7f158013f710 legacy=0x7f15801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 >> v1:172.21.15.176:6789/0 conn(0x7f15801430c0 legacy=0x7f158014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:41.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 monclient: found mon.noname-c 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15800b44c0 con 0x7f15800a98f0 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f157c001690 con 0x7f15800a98f0 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f15874af640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f15874af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f1577fff640 10 monclient: dump: 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.729 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.727+0000 7f15874af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f157c0017f0 con 0x7f15800a98f0 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 1 -- 172.21.15.176:0/186038042 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f157c001c00 con 0x7f15800a98f0 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:41.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f1577fff640 10 monclient: dump: 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f15874af640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:41.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.735+0000 7f15874af640 4 set_mon_vals no callback set 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 10 monclient: shutdown 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 1 -- 172.21.15.176:0/186038042 >> v1:172.21.15.186:6789/0 conn(0x7f15800a98f0 legacy=0x7f15800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 1 -- 172.21.15.176:0/186038042 shutdown_connections 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 1 -- 172.21.15.176:0/186038042 >> 172.21.15.176:0/186038042 conn(0x7f15800b7ea0 msgr2=0x7f15800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.736+0000 7f15874af640 1 -- 172.21.15.176:0/186038042 shutdown_connections 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.737+0000 7f15874af640 1 -- 172.21.15.176:0/186038042 wait complete. 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.737+0000 7f15874af640 10 monclient: build_initial_monmap 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.737+0000 7f15874af640 10 monclient: monmap: 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:41.740 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 librados: starting msgr at 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 librados: starting objecter 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 Processor -- start 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 -- start start 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 librados: setting wanted keys 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 1 librados: calling monclient init 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 10 monclient: init 2024-03-25T18:25:41.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.738+0000 7f15874af640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient(hunting): picked mon.noname-c con 0x7f15800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient(hunting): picked mon.noname-b con 0x7f158013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient(hunting): picked mon.noname-a con 0x7f15801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580135730 con 0x7f15801430c0 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15801358c0 con 0x7f158013f710 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580135a50 con 0x7f15800a98f0 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:41.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.739+0000 7f15874af640 10 monclient(hunting): authenticate will time out at 8220.050935s 2024-03-25T18:25:41.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.743+0000 7f158524b640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f15800a98f0 0x7f15801dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48894/0 (socket says 172.21.15.176:48894) 2024-03-25T18:25:41.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.743+0000 7f158524b640 1 -- 172.21.15.176:0/2479769127 learned_addr learned my addr 172.21.15.176:0/2479769127 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:41.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.744+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f15801430c0 2024-03-25T18:25:41.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.744+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.745+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f15800a98f0 2024-03-25T18:25:41.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.745+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532887594 0 0) 0x7f1580135730 con 0x7f15801430c0 2024-03-25T18:25:41.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:25:41.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.759+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564004eb0 con 0x7f15801430c0 2024-03-25T18:25:41.763 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.762+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f158013f710 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.762+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993924265 0 0) 0x7f1580135a50 con 0x7f15800a98f0 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:25:41.764 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.763+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580135730 con 0x7f15800a98f0 2024-03-25T18:25:41.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.776+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437007118 0 0) 0x7f15801358c0 con 0x7f158013f710 2024-03-25T18:25:41.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.776+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:41.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.776+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:41.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.777+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:41.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.777+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:25:41.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.777+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580135a50 con 0x7f158013f710 2024-03-25T18:25:41.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385946271 0 0) 0x7f1564004eb0 con 0x7f15801430c0 2024-03-25T18:25:41.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15801358c0 con 0x7f15801430c0 2024-03-25T18:25:41.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1578001bd0 con 0x7f15801430c0 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:41.782 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:41.783 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:41.783 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.780+0000 7f1575ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4283758112 0 0) 0x7f1580135730 con 0x7f15800a98f0 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1564004eb0 con 0x7f15800a98f0 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f157c001b40 con 0x7f15800a98f0 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.788+0000 7f1575ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:41.789 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:41.790 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:41.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3566582474 0 0) 0x7f1564004eb0 con 0x7f15800a98f0 2024-03-25T18:25:41.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 >> v1:172.21.15.186:6789/0 conn(0x7f158013f710 legacy=0x7f15801d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 >> v1:172.21.15.176:6789/0 conn(0x7f15801430c0 legacy=0x7f15801d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 1 monclient: found mon.c 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15801d5000 con 0x7f15800a98f0 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f1575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 10 monclient: _renew_subs 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:41.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15801d52b0 con 0x7f15800a98f0 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 10 monclient: _renew_subs 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f158013c900 con 0x7f15800a98f0 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.802+0000 7f15874af640 1 librados: init done 2024-03-25T18:25:41.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.803+0000 7f15874af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:41.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.803+0000 7f15874af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:41.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.803+0000 7f15874af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:41.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:41.803+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15801371c0 con 0x7f15800a98f0 2024-03-25T18:25:42.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f157c004a90 con 0x7f15800a98f0 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f157c004da0 con 0x7f15800a98f0 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 10 monclient: dump: 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:42.108 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f15777fe640 4 set_mon_vals no callback set 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.107+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f157c01ad70 con 0x7f15800a98f0 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.108+0000 7f1584a4a640 10 monclient: get_auth_request con 0x7f156406dfb0 auth_method 0 2024-03-25T18:25:42.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.108+0000 7f1575ffb640 10 client.4289.objecter ms_handle_connect 0x7f156406dfb0 2024-03-25T18:25:42.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f157c0548c0 con 0x7f15800a98f0 2024-03-25T18:25:42.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 10 client.4289.objecter ms_dispatch 0x7f1580000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:42.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:42.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:42.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T18:25:42.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f157c0503d0 con 0x7f15800a98f0 2024-03-25T18:25:42.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:42.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.114+0000 7f1575ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:42.339 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.338+0000 7f15874af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.340 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.338+0000 7f15874af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.340 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.338+0000 7f15874af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:25:42.340 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.338+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15800627b0 con 0x7f15800a98f0 2024-03-25T18:25:42.433 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:44.189 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.444 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:44.200 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.494 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:44.228 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.497 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:01:44.253 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.500 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:44.235 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.513+0000 7f1575ffb640 1 -- 172.21.15.176:0/2479769127 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f157c052300 con 0x7f15800a98f0 2024-03-25T18:25:42.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.514+0000 7f1575ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:42.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.514+0000 7f1575ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:42.515 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:42.515 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:42.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 10 librados: watch_flush enter 2024-03-25T18:25:42.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 10 librados: watch_flush exit 2024-03-25T18:25:42.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T18:25:42.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 10 client.4289.objecter successfully canceled tick 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 >> v1:172.21.15.176:6800/2598213983 conn(0x7f156406dfb0 legacy=0x7f1564070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 10 monclient: shutdown 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 >> v1:172.21.15.176:6790/0 conn(0x7f15800a98f0 legacy=0x7f15801dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 shutdown_connections 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 >> 172.21.15.176:0/2479769127 conn(0x7f15800b7ea0 msgr2=0x7f158014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 shutdown_connections 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 -- 172.21.15.176:0/2479769127 wait complete. 2024-03-25T18:25:42.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:42.516+0000 7f15874af640 1 librados: shutdown 2024-03-25T18:25:42.519 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:44.275 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.525 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:44.282 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:42.553 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:44.288 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.582 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:01:44.338 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:49.334 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.446+0000 7fdd5f7e6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.446+0000 7fdd5f7e6640 10 monclient: build_initial_monmap 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.446+0000 7fdd5f7e6640 10 monclient: monmap: 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:49.447273+0000 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:49.447273+0000 2024-03-25T18:25:49.447 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:49.448 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.448 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:49.448 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:49.448 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:49.448 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 1 Processor -- start 2024-03-25T18:25:49.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 1 -- start start 2024-03-25T18:25:49.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient: init 2024-03-25T18:25:49.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:49.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd580a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.447+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd58142520 con 0x7fdd5813f710 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.448+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd580b28a0 con 0x7fdd580a98f0 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.448+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd580b2a30 con 0x7fdd581430c0 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.448+0000 7fdd5f7e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:49.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.448+0000 7fdd5f7e6640 10 monclient(hunting): authenticate will time out at 8227.759866s 2024-03-25T18:25:49.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.453+0000 7fdd5dd83640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fdd581430c0 0x7fdd5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53492/0 (socket says 172.21.15.176:53492) 2024-03-25T18:25:49.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.453+0000 7fdd5dd83640 1 -- 172.21.15.176:0/3058177540 learned_addr learned my addr 172.21.15.176:0/3058177540 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:49.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799740110 0 0) 0x7fdd580b28a0 con 0x7fdd580a98f0 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd44003520 con 0x7fdd580a98f0 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157721188 0 0) 0x7fdd58142520 con 0x7fdd5813f710 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:25:49.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.474+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd580b28a0 con 0x7fdd5813f710 2024-03-25T18:25:49.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851614708 0 0) 0x7fdd580b2a30 con 0x7fdd581430c0 2024-03-25T18:25:49.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:25:49.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.480+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58142520 con 0x7fdd581430c0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207887982 0 0) 0x7fdd44003520 con 0x7fdd580a98f0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 >> v1:172.21.15.176:6790/0 conn(0x7fdd581430c0 legacy=0x7fdd5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 >> v1:172.21.15.176:6789/0 conn(0x7fdd5813f710 legacy=0x7fdd581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 monclient: found mon.noname-c 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580b44c0 con 0x7fdd580a98f0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd54001690 con 0x7fdd580a98f0 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd4ffff640 10 monclient: dump: 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd5f7e6640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:25:49.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.491+0000 7fdd5f7e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:49.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd540017f0 con 0x7fdd580a98f0 2024-03-25T18:25:49.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:49.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 1 -- 172.21.15.176:0/3058177540 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd54001c00 con 0x7fdd580a98f0 2024-03-25T18:25:49.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:49.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 4 set_mon_vals no callback set 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd4ffff640 10 monclient: dump: 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:49.495 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 10 monclient: shutdown 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/3058177540 >> v1:172.21.15.186:6789/0 conn(0x7fdd580a98f0 legacy=0x7fdd580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/3058177540 shutdown_connections 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.494+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/3058177540 >> 172.21.15.176:0/3058177540 conn(0x7fdd580b7ea0 msgr2=0x7fdd580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/3058177540 shutdown_connections 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/3058177540 wait complete. 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 10 monclient: build_initial_monmap 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 10 monclient: monmap: 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:49.498 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 1 librados: starting msgr at 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.495+0000 7fdd5f7e6640 1 librados: starting objecter 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 Processor -- start 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 -- start start 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 librados: setting wanted keys 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 librados: calling monclient init 2024-03-25T18:25:49.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient: init 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd580a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd5813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd581430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5814c6b0 con 0x7fdd581430c0 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd581349e0 con 0x7fdd5813f710 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5814d9b0 con 0x7fdd580a98f0 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:49.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.496+0000 7fdd5f7e6640 10 monclient(hunting): authenticate will time out at 8227.808675s 2024-03-25T18:25:49.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.506+0000 7fdd5dd83640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdd581430c0 0x7fdd581342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:40200/0 (socket says 172.21.15.176:40200) 2024-03-25T18:25:49.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.506+0000 7fdd5dd83640 1 -- 172.21.15.176:0/755117442 learned_addr learned my addr 172.21.15.176:0/755117442 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.507+0000 7fdd4dffb640 10 client.?.objecter ms_handle_connect 0x7fdd581430c0 2024-03-25T18:25:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.507+0000 7fdd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 client.?.objecter ms_handle_connect 0x7fdd580a98f0 2024-03-25T18:25:49.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd5d582640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6790/0 conn(0x7fdd580a98f0 legacy=0x7fdd580b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd5d582640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6790/0 conn(0x7fdd580a98f0 legacy=0x7fdd580b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd5d582640 1 --1- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6790/0 conn(0x7fdd580a98f0 0x7fdd580b7630 :-1 s=OPENED pgs=5552 cs=1 l=1).handle_message read tag failed 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd5d582640 1 --1- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6790/0 conn(0x7fdd580a98f0 0x7fdd580b7630 :-1 s=OPENED pgs=5552 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6790/0 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547773536 0 0) 0x7fdd5814c6b0 con 0x7fdd581430c0 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:25:49.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.516+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3c004eb0 con 0x7fdd581430c0 2024-03-25T18:25:49.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.519+0000 7fdd4dffb640 10 client.?.objecter ms_handle_connect 0x7fdd5813f710 2024-03-25T18:25:49.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.519+0000 7fdd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020910732 0 0) 0x7fdd581349e0 con 0x7fdd5813f710 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:25:49.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.524+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5814c6b0 con 0x7fdd5813f710 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245287876 0 0) 0x7fdd3c004eb0 con 0x7fdd581430c0 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd581349e0 con 0x7fdd581430c0 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd50001bd0 con 0x7fdd581430c0 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:49.547 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.546+0000 7fdd4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58893794 0 0) 0x7fdd5814c6b0 con 0x7fdd5813f710 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd3c004eb0 con 0x7fdd5813f710 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd48001930 con 0x7fdd5813f710 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.555+0000 7fdd4dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.556 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:49.557 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:49.557 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:49.557 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656289285 0 0) 0x7fdd581349e0 con 0x7fdd581430c0 2024-03-25T18:25:49.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6790/0 conn(0x7fdd580a98f0 legacy=0x7fdd580b7630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.186:6789/0 conn(0x7fdd5813f710 legacy=0x7fdd58133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 1 monclient: found mon.a 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5814ca00 con 0x7fdd581430c0 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 10 monclient: _renew_subs 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:49.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5814cc50 con 0x7fdd581430c0 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 10 monclient: _renew_subs 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5813b3a0 con 0x7fdd581430c0 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.568+0000 7fdd5f7e6640 1 librados: init done 2024-03-25T18:25:49.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.569+0000 7fdd5f7e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.569+0000 7fdd5f7e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.569+0000 7fdd5f7e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:49.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.569+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd580627b0 con 0x7fdd581430c0 2024-03-25T18:25:49.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.578+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd50002080 con 0x7fdd581430c0 2024-03-25T18:25:49.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.579+0000 7fdd4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:49.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.579+0000 7fdd4f7fe640 4 set_mon_vals no callback set 2024-03-25T18:25:49.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.871+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdd50004d40 con 0x7fdd581430c0 2024-03-25T18:25:49.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.871+0000 7fdd4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:49.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.871+0000 7fdd4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:49.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.871+0000 7fdd4dffb640 10 monclient: dump: 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:49.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.871+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fdd50004ea0 con 0x7fdd581430c0 2024-03-25T18:25:49.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.872+0000 7fdd5d582640 10 monclient: get_auth_request con 0x7fdd3c06de70 auth_method 0 2024-03-25T18:25:49.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.872+0000 7fdd4dffb640 10 client.4278.objecter ms_handle_connect 0x7fdd3c06de70 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fdd50054a80 con 0x7fdd581430c0 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 10 client.4278.objecter ms_dispatch 0x7fdd58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd500503c0 con 0x7fdd581430c0 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:49.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:49.877+0000 7fdd4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:50.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.107+0000 7fdd5f7e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.107+0000 7fdd5f7e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.107+0000 7fdd5f7e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:50.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.107+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd5813df70 con 0x7fdd581430c0 2024-03-25T18:25:50.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.148+0000 7fdd4dffb640 1 -- 172.21.15.176:0/755117442 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fdd500524c0 con 0x7fdd581430c0 2024-03-25T18:25:50.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.148+0000 7fdd4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.148+0000 7fdd4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:50.150 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:50.150 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:50.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 10 librados: watch_flush enter 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 10 librados: watch_flush exit 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 10 client.4278.objecter successfully canceled tick 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6800/2598213983 conn(0x7fdd3c06de70 legacy=0x7fdd3c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 10 monclient: shutdown 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 >> v1:172.21.15.176:6789/0 conn(0x7fdd581430c0 legacy=0x7fdd581342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 shutdown_connections 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.151+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 >> 172.21.15.176:0/755117442 conn(0x7fdd580b7ea0 msgr2=0x7fdd58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.152+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 shutdown_connections 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.152+0000 7fdd5f7e6640 1 -- 172.21.15.176:0/755117442 wait complete. 2024-03-25T18:25:50.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:50.152+0000 7fdd5f7e6640 1 librados: shutdown 2024-03-25T18:25:50.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:56.972 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:57.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.085+0000 7f354c84a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.085+0000 7f354c84a640 10 monclient: build_initial_monmap 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.085+0000 7f354c84a640 10 monclient: monmap: 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:25:57.086611+0000 2024-03-25T18:25:57.089 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:25:57.086611+0000 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:25:57.090 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.086+0000 7f354c84a640 1 Processor -- start 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.086+0000 7f354c84a640 1 -- start start 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.086+0000 7f354c84a640 10 monclient: init 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-a con 0x7f3548061830 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-c con 0x7f354813db20 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-b con 0x7f35480b0510 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354806eba0 con 0x7f3548061830 2024-03-25T18:25:57.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480b0960 con 0x7f354813db20 2024-03-25T18:25:57.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548072830 con 0x7f35480b0510 2024-03-25T18:25:57.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:57.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.087+0000 7f354c84a640 10 monclient(hunting): authenticate will time out at 8235.399472s 2024-03-25T18:25:57.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.094+0000 7f3546d9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 0x7f354814ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:46864/0 (socket says 172.21.15.176:46864) 2024-03-25T18:25:57.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.094+0000 7f3546d9d640 1 -- 172.21.15.176:0/2640974271 learned_addr learned my addr 172.21.15.176:0/2640974271 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:57.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486809029 0 0) 0x7f3548072830 con 0x7f35480b0510 2024-03-25T18:25:57.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:25:57.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.097+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530003520 con 0x7f35480b0510 2024-03-25T18:25:57.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936680544 0 0) 0x7f35480b0960 con 0x7f354813db20 2024-03-25T18:25:57.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:25:57.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.106+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3548072830 con 0x7f354813db20 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207003843 0 0) 0x7f3530003520 con 0x7f35480b0510 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 >> v1:172.21.15.186:6789/0 conn(0x7f354813db20 legacy=0x7f35480afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 >> v1:172.21.15.176:6789/0 conn(0x7f3548061830 legacy=0x7f354813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 monclient: found mon.noname-b 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548072b10 con 0x7f35480b0510 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: _finish_auth 0 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:57.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3540001a10 con 0x7f35480b0510 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354c84a640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354c84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354559a640 10 monclient: dump: 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:57.115 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:57.116 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:57.116 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.112+0000 7f354c84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:57.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3540001b70 con 0x7f35480b0510 2024-03-25T18:25:57.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:57.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354559a640 1 -- 172.21.15.176:0/2640974271 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3540001f00 con 0x7f35480b0510 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354c84a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354c84a640 4 set_mon_vals no callback set 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.139+0000 7f354559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354559a640 10 monclient: dump: 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:57.141 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:57.142 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 10 monclient: shutdown 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 1 -- 172.21.15.176:0/2640974271 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 legacy=0x7f354814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 1 -- 172.21.15.176:0/2640974271 shutdown_connections 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 1 -- 172.21.15.176:0/2640974271 >> 172.21.15.176:0/2640974271 conn(0x7f35480b0ef0 msgr2=0x7f35480b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 1 -- 172.21.15.176:0/2640974271 shutdown_connections 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.140+0000 7f354c84a640 1 -- 172.21.15.176:0/2640974271 wait complete. 2024-03-25T18:25:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.141+0000 7f354c84a640 10 monclient: build_initial_monmap 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.141+0000 7f354c84a640 10 monclient: monmap: 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:25:57.144 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 librados: starting msgr at 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 librados: starting objecter 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 Processor -- start 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 -- start start 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 librados: setting wanted keys 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 1 librados: calling monclient init 2024-03-25T18:25:57.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.142+0000 7f354c84a640 10 monclient: init 2024-03-25T18:25:57.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:57.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:57.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-b con 0x7f3548061830 addr v1:172.21.15.186:6789/0 2024-03-25T18:25:57.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-c con 0x7f35480b0510 addr v1:172.21.15.176:6790/0 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient(hunting): picked mon.noname-a con 0x7f354813db20 addr v1:172.21.15.176:6789/0 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354813ad60 con 0x7f354813db20 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548138e70 con 0x7f3548061830 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548139e00 con 0x7f35480b0510 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.143+0000 7f354c84a640 10 monclient(hunting): authenticate will time out at 8235.455124s 2024-03-25T18:25:57.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.147+0000 7f354659c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3548061830 0x7f35481d5b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:38742/0 (socket says 172.21.15.176:38742) 2024-03-25T18:25:57.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.147+0000 7f354659c640 1 -- 172.21.15.176:0/2988641783 learned_addr learned my addr 172.21.15.176:0/2988641783 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:25:57.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.150+0000 7f3536ffd640 10 client.?.objecter ms_handle_connect 0x7f3548061830 2024-03-25T18:25:57.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.150+0000 7f3536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:57.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.155+0000 7f3536ffd640 10 client.?.objecter ms_handle_connect 0x7f354813db20 2024-03-25T18:25:57.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.155+0000 7f3536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:57.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.157+0000 7f3536ffd640 10 client.?.objecter ms_handle_connect 0x7f35480b0510 2024-03-25T18:25:57.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.157+0000 7f3536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:57.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.158+0000 7f3545d9b640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 legacy=0x7f35481d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:25:57.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.158+0000 7f3545d9b640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 legacy=0x7f35481d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:25:57.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.158+0000 7f3545d9b640 1 --1- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 0x7f35481d1ed0 :-1 s=OPENED pgs=6591 cs=1 l=1).handle_message read tag failed 2024-03-25T18:25:57.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.158+0000 7f3545d9b640 1 --1- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 0x7f35481d1ed0 :-1 s=OPENED pgs=6591 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:25:57.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.158+0000 7f3536ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6790/0 2024-03-25T18:25:57.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072072888 0 0) 0x7f3548138e70 con 0x7f3548061830 2024-03-25T18:25:57.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:25:57.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.163+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3528004eb0 con 0x7f3548061830 2024-03-25T18:25:57.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284712438 0 0) 0x7f354813ad60 con 0x7f354813db20 2024-03-25T18:25:57.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:25:57.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.166+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3548138e70 con 0x7f354813db20 2024-03-25T18:25:57.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.176+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151322024 0 0) 0x7f3548138e70 con 0x7f354813db20 2024-03-25T18:25:57.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.176+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f354813ad60 con 0x7f354813db20 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3989679545 0 0) 0x7f3528004eb0 con 0x7f3548061830 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3548138e70 con 0x7f3548061830 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3538001a30 con 0x7f3548061830 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.182 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3540001ce0 con 0x7f354813db20 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.181+0000 7f3536ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.183 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.184 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:57.184 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:57.184 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:57.184 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1066821930 0 0) 0x7f354813ad60 con 0x7f354813db20 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6790/0 conn(0x7f35480b0510 legacy=0x7f35481d1ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.186:6789/0 conn(0x7f3548061830 legacy=0x7f35481d5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 1 monclient: found mon.a 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548139540 con 0x7f354813db20 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f3536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 10 monclient: _renew_subs 2024-03-25T18:25:57.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35480ad960 con 0x7f354813db20 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 10 monclient: _renew_subs 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.198+0000 7f354c84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.199+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35481d2ed0 con 0x7f354813db20 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.199+0000 7f354c84a640 1 librados: init done 2024-03-25T18:25:57.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.200+0000 7f354c84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.200+0000 7f354c84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.200+0000 7f354c84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:57.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.200+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3548139060 con 0x7f354813db20 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3540001e40 con 0x7f354813db20 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3540004fb0 con 0x7f354813db20 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:57.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3536ffd640 10 monclient: dump: 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:25:57.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.506+0000 7f3544d99640 4 set_mon_vals no callback set 2024-03-25T18:25:57.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.507+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f354001af80 con 0x7f354813db20 2024-03-25T18:25:57.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.508+0000 7f354659c640 10 monclient: get_auth_request con 0x7f352806deb0 auth_method 0 2024-03-25T18:25:57.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.508+0000 7f3536ffd640 10 client.4284.objecter ms_handle_connect 0x7f352806deb0 2024-03-25T18:25:57.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f3540054a40 con 0x7f354813db20 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 10 client.4284.objecter ms_dispatch 0x7f3548000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3540050550 con 0x7f354813db20 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.514+0000 7f3536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:57.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.746+0000 7f354c84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.746+0000 7f354c84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.746+0000 7f354c84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:25:57.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.746+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35480627b0 con 0x7f354813db20 2024-03-25T18:25:57.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.769+0000 7f3536ffd640 1 -- 172.21.15.176:0/2988641783 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f3540052480 con 0x7f354813db20 2024-03-25T18:25:57.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.769+0000 7f3536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:57.770 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.769+0000 7f3536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:57.770 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:25:57.770 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.770+0000 7f354c84a640 10 librados: watch_flush enter 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 10 librados: watch_flush exit 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 10 client.4284.objecter successfully canceled tick 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6800/2598213983 conn(0x7f352806deb0 legacy=0x7f3528070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 10 monclient: shutdown 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:57.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 >> v1:172.21.15.176:6789/0 conn(0x7f354813db20 legacy=0x7f35481d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 shutdown_connections 2024-03-25T18:25:57.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 >> 172.21.15.176:0/2988641783 conn(0x7f35480b0ef0 msgr2=0x7f35480afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:57.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 shutdown_connections 2024-03-25T18:25:57.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 -- 172.21.15.176:0/2988641783 wait complete. 2024-03-25T18:25:57.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:25:57.771+0000 7f354c84a640 1 librados: shutdown 2024-03-25T18:25:57.782 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:02.642 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:04.397 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.655 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:04.411 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.696 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:04.453 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.698 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:04.454 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.701 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:04.436 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.707 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:04.442 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.708 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:04.464 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.732 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:04.488 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.738 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:04.472 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.590 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.700+0000 7f0b51f27640 10 monclient: get_monmap_and_config 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.700+0000 7f0b51f27640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.700+0000 7f0b51f27640 10 monclient: monmap: 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:04.701330+0000 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:04.701330+0000 2024-03-25T18:26:04.701 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.702 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.702 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:04.702 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:04.702 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:04.702 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.701+0000 7f0b51f27640 1 Processor -- start 2024-03-25T18:26:04.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.701+0000 7f0b51f27640 1 -- start start 2024-03-25T18:26:04.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.701+0000 7f0b51f27640 10 monclient: init 2024-03-25T18:26:04.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.701+0000 7f0b51f27640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.701+0000 7f0b51f27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-b con 0x7f0b4c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-c con 0x7f0b4c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-a con 0x7f0b4c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c142520 con 0x7f0b4c1430c0 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c0b28a0 con 0x7f0b4c13f710 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c0b2a30 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.702+0000 7f0b51f27640 10 monclient(hunting): authenticate will time out at 8243.014014s 2024-03-25T18:26:04.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.711+0000 7f0b4bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0b4c1430c0 0x7f0b4c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:44610/0 (socket says 172.21.15.176:44610) 2024-03-25T18:26:04.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.711+0000 7f0b4bfff640 1 -- 172.21.15.176:0/2278321194 learned_addr learned my addr 172.21.15.176:0/2278321194 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:04.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.711+0000 7f0b4b7fe640 1 --1- 172.21.15.176:0/2278321194 >> v1:172.21.15.176:6790/0 conn(0x7f0b4c0a98f0 0x7f0b4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:40736/0 (socket says 172.21.15.176:40736) 2024-03-25T18:26:04.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994763925 0 0) 0x7f0b4c0b2a30 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:26:04.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.727+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b34003520 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228420910 0 0) 0x7f0b4c0b28a0 con 0x7f0b4c13f710 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:26:04.730 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.729+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c0b2a30 con 0x7f0b4c13f710 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200664293 0 0) 0x7f0b4c142520 con 0x7f0b4c1430c0 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:26:04.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.734+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c0b28a0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2360729157 0 0) 0x7f0b4c0b2a30 con 0x7f0b4c13f710 2024-03-25T18:26:04.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 >> v1:172.21.15.176:6790/0 conn(0x7f0b4c0a98f0 legacy=0x7f0b4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 >> v1:172.21.15.176:6789/0 conn(0x7f0b4c1430c0 legacy=0x7f0b4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c0b44c0 con 0x7f0b4c13f710 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b40001670 con 0x7f0b4c13f710 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b4a7fc640 10 monclient: dump: 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.753 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b51f27640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:26:04.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.751+0000 7f0b51f27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b400017d0 con 0x7f0b4c13f710 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 1 -- 172.21.15.176:0/2278321194 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b40001be0 con 0x7f0b4c13f710 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b51f27640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b51f27640 4 set_mon_vals no callback set 2024-03-25T18:26:04.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.774+0000 7f0b4a7fc640 10 monclient: dump: 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:04.776 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:04.777 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 10 monclient: shutdown 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 1 -- 172.21.15.176:0/2278321194 >> v1:172.21.15.186:6789/0 conn(0x7f0b4c13f710 legacy=0x7f0b4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 1 -- 172.21.15.176:0/2278321194 shutdown_connections 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 1 -- 172.21.15.176:0/2278321194 >> 172.21.15.176:0/2278321194 conn(0x7f0b4c0b7ea0 msgr2=0x7f0b4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 1 -- 172.21.15.176:0/2278321194 shutdown_connections 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.775+0000 7f0b51f27640 1 -- 172.21.15.176:0/2278321194 wait complete. 2024-03-25T18:26:04.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 10 monclient: monmap: 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 librados: starting msgr at 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 librados: starting objecter 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 Processor -- start 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 -- start start 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 librados: setting wanted keys 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 1 librados: calling monclient init 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.776+0000 7f0b51f27640 10 monclient: init 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-a con 0x7f0b4c0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-c con 0x7f0b4c13f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient(hunting): picked mon.noname-b con 0x7f0b4c1430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c0abde0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c0ac0c0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c139250 con 0x7f0b4c13f710 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.777+0000 7f0b51f27640 10 monclient(hunting): authenticate will time out at 8243.089476s 2024-03-25T18:26:04.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.789+0000 7f0b4affd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f0b4c13f710 0x7f0b4c0ac4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:40798/0 (socket says 172.21.15.176:40798) 2024-03-25T18:26:04.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.789+0000 7f0b4affd640 1 -- 172.21.15.176:0/1217420408 learned_addr learned my addr 172.21.15.176:0/1217420408 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:04.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.789+0000 7f0b33fff640 10 client.?.objecter ms_handle_connect 0x7f0b4c13f710 2024-03-25T18:26:04.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.789+0000 7f0b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.790+0000 7f0b33fff640 10 client.?.objecter ms_handle_connect 0x7f0b4c1430c0 2024-03-25T18:26:04.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.790+0000 7f0b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.793+0000 7f0b33fff640 10 client.?.objecter ms_handle_connect 0x7f0b4c0a98f0 2024-03-25T18:26:04.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.793+0000 7f0b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435796468 0 0) 0x7f0b4c0ac0c0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:26:04.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.800+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b38004eb0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300101672 0 0) 0x7f0b4c139250 con 0x7f0b4c13f710 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:26:04.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.804+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c0ac0c0 con 0x7f0b4c13f710 2024-03-25T18:26:04.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.807+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61748574 0 0) 0x7f0b4c0abde0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.807+0000 7f0b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.807+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.807+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.808+0000 7f0b33fff640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:26:04.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.808+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c139250 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.816+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655326412 0 0) 0x7f0b4c139250 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.816+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b4c0abde0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.816+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b3c0017b0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.816+0000 7f0b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): dump: 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.818 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022501297 0 0) 0x7f0b38004eb0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b4c139250 con 0x7f0b4c1430c0 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b44001bd0 con 0x7f0b4c1430c0 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.817+0000 7f0b33fff640 10 monclient(hunting): dump: 2024-03-25T18:26:04.819 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:04.820 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015724777 0 0) 0x7f0b4c0ac0c0 con 0x7f0b4c13f710 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b38004eb0 con 0x7f0b4c13f710 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b40001670 con 0x7f0b4c13f710 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.823+0000 7f0b33fff640 10 monclient(hunting): dump: 2024-03-25T18:26:04.824 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:04.825 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:04.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2426524232 0 0) 0x7f0b4c0abde0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 >> v1:172.21.15.176:6790/0 conn(0x7f0b4c13f710 legacy=0x7f0b4c0ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 >> v1:172.21.15.186:6789/0 conn(0x7f0b4c1430c0 legacy=0x7f0b4c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 1 monclient: found mon.a 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c0acbf0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 10 monclient: _renew_subs 2024-03-25T18:26:04.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c137de0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 10 monclient: _renew_subs 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c1368c0 con 0x7f0b4c0a98f0 2024-03-25T18:26:04.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.827+0000 7f0b51f27640 1 librados: init done 2024-03-25T18:26:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.828+0000 7f0b51f27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.828+0000 7f0b51f27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.828+0000 7f0b51f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:04.828+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c136bc0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.180+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b3c001cc0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b49ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b3c0047e0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 10 monclient: dump: 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:05.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.181+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f0b3c01a7b0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.182+0000 7f0b4affd640 10 monclient: get_auth_request con 0x7f0b3806e0b0 auth_method 0 2024-03-25T18:26:05.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.182+0000 7f0b33fff640 10 client.4290.objecter ms_handle_connect 0x7f0b3806e0b0 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f0b3c054520 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 10 client.4290.objecter ms_dispatch 0x7f0b4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b3c04fee0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.187+0000 7f0b33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:05.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.418+0000 7f0b51f27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:05.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.418+0000 7f0b51f27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:05.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.418+0000 7f0b51f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:05.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.418+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b4c14f2d0 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.440+0000 7f0b33fff640 1 -- 172.21.15.176:0/1217420408 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f0b3c051f60 con 0x7f0b4c0a98f0 2024-03-25T18:26:05.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.440+0000 7f0b33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:05.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.440+0000 7f0b33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:05.441 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:05.442 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 10 librados: watch_flush enter 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 10 librados: watch_flush exit 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 10 client.4290.objecter successfully canceled tick 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 >> v1:172.21.15.176:6800/2598213983 conn(0x7f0b3806e0b0 legacy=0x7f0b38070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 10 monclient: shutdown 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 >> v1:172.21.15.176:6789/0 conn(0x7f0b4c0a98f0 legacy=0x7f0b4c138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 shutdown_connections 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 >> 172.21.15.176:0/1217420408 conn(0x7f0b4c0b7ea0 msgr2=0x7f0b4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 shutdown_connections 2024-03-25T18:26:05.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 -- 172.21.15.176:0/1217420408 wait complete. 2024-03-25T18:26:05.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:05.443+0000 7f0b51f27640 1 librados: shutdown 2024-03-25T18:26:05.456 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:12.263 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:12.377 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.376+0000 7f145d38c640 10 monclient: get_monmap_and_config 2024-03-25T18:26:12.377 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.376+0000 7f145d38c640 10 monclient: build_initial_monmap 2024-03-25T18:26:12.377 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.376+0000 7f145d38c640 10 monclient: monmap: 2024-03-25T18:26:12.377 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:12.377705+0000 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:12.377705+0000 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:12.378 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.377+0000 7f145d38c640 1 Processor -- start 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.377+0000 7f145d38c640 1 -- start start 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.377+0000 7f145d38c640 10 monclient: init 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-c con 0x7f14580a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-a con 0x7f145813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-b con 0x7f14581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1458142520 con 0x7f145813f710 2024-03-25T18:26:12.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580b28a0 con 0x7f14580a98f0 2024-03-25T18:26:12.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580b2a30 con 0x7f14581430c0 2024-03-25T18:26:12.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:12.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.378+0000 7f145d38c640 10 monclient(hunting): authenticate will time out at 8250.690120s 2024-03-25T18:26:12.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.379+0000 7f1456ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f14580a98f0 0x7f14580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:59632/0 (socket says 172.21.15.176:59632) 2024-03-25T18:26:12.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.379+0000 7f1456ffd640 1 -- 172.21.15.176:0/2475132687 learned_addr learned my addr 172.21.15.176:0/2475132687 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580501937 0 0) 0x7f14580b28a0 con 0x7f14580a98f0 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:26:12.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.392+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1440003520 con 0x7f14580a98f0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959885724 0 0) 0x7f1440003520 con 0x7f14580a98f0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 >> v1:172.21.15.176:6790/0 conn(0x7f14581430c0 legacy=0x7f145814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 >> v1:172.21.15.176:6789/0 conn(0x7f145813f710 legacy=0x7f14581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 monclient: found mon.noname-c 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580b44c0 con 0x7f14580a98f0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:12.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1448001690 con 0x7f14580a98f0 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f1455ffb640 10 monclient: dump: 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:12.414 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f145d38c640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:26:12.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.412+0000 7f145d38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:12.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.417+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14480017f0 con 0x7f14580a98f0 2024-03-25T18:26:12.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f1455ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:12.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f1455ffb640 1 -- 172.21.15.176:0/2475132687 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1448001c00 con 0x7f14580a98f0 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f1455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f1455ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f1455ffb640 10 monclient: dump: 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f145d38c640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:12.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f145d38c640 4 set_mon_vals no callback set 2024-03-25T18:26:12.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f145d38c640 10 monclient: shutdown 2024-03-25T18:26:12.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f145d38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:12.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.418+0000 7f145d38c640 1 -- 172.21.15.176:0/2475132687 >> v1:172.21.15.186:6789/0 conn(0x7f14580a98f0 legacy=0x7f14580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.419+0000 7f145d38c640 1 -- 172.21.15.176:0/2475132687 shutdown_connections 2024-03-25T18:26:12.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.419+0000 7f145d38c640 1 -- 172.21.15.176:0/2475132687 >> 172.21.15.176:0/2475132687 conn(0x7f14580b7ea0 msgr2=0x7f14580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.419+0000 7f145d38c640 1 -- 172.21.15.176:0/2475132687 shutdown_connections 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.419+0000 7f145d38c640 1 -- 172.21.15.176:0/2475132687 wait complete. 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 10 monclient: build_initial_monmap 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 10 monclient: monmap: 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:12.421 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 librados: starting msgr at 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 librados: starting objecter 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 Processor -- start 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 -- start start 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 librados: setting wanted keys 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 1 librados: calling monclient init 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.420+0000 7f145d38c640 10 monclient: init 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:12.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-a con 0x7f14580a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-b con 0x7f145813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient(hunting): picked mon.noname-c con 0x7f14581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1458137370 con 0x7f14580a98f0 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14581352a0 con 0x7f145813f710 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1458131760 con 0x7f14581430c0 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:12.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.421+0000 7f145d38c640 10 monclient(hunting): authenticate will time out at 8250.733499s 2024-03-25T18:26:12.429 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.427+0000 7f1456ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f14580a98f0 0x7f1458131050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49650/0 (socket says 172.21.15.176:49650) 2024-03-25T18:26:12.429 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.427+0000 7f1456ffd640 1 -- 172.21.15.176:0/267221681 learned_addr learned my addr 172.21.15.176:0/267221681 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:12.429 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.428+0000 7f143f7fe640 10 client.?.objecter ms_handle_connect 0x7f14580a98f0 2024-03-25T18:26:12.430 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.428+0000 7f143f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.433 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.433+0000 7f143f7fe640 10 client.?.objecter ms_handle_connect 0x7f145813f710 2024-03-25T18:26:12.433 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.433+0000 7f143f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.435+0000 7f143f7fe640 10 client.?.objecter ms_handle_connect 0x7f14581430c0 2024-03-25T18:26:12.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.435+0000 7f143f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228851100 0 0) 0x7f1458131760 con 0x7f14581430c0 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:26:12.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1444004eb0 con 0x7f14581430c0 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702177313 0 0) 0x7f1458137370 con 0x7f14580a98f0 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:26:12.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.443+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458131760 con 0x7f14580a98f0 2024-03-25T18:26:12.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258316416 0 0) 0x7f14581352a0 con 0x7f145813f710 2024-03-25T18:26:12.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:12.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:12.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:12.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:26:12.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.449+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458137370 con 0x7f145813f710 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359246674 0 0) 0x7f1458131760 con 0x7f14580a98f0 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14581352a0 con 0x7f14580a98f0 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1448001b40 con 0x7f14580a98f0 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:12.463 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.461+0000 7f143f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813396577 0 0) 0x7f1444004eb0 con 0x7f14581430c0 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1458131760 con 0x7f14581430c0 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1450001b90 con 0x7f14581430c0 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.462+0000 7f143f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:12.464 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:12.465 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:12.465 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165608769 0 0) 0x7f1458131760 con 0x7f14581430c0 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 >> v1:172.21.15.186:6789/0 conn(0x7f145813f710 legacy=0x7f14581d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 >> v1:172.21.15.176:6789/0 conn(0x7f14580a98f0 legacy=0x7f1458131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 1 monclient: found mon.c 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14581318f0 con 0x7f14581430c0 2024-03-25T18:26:12.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f143f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 10 monclient: _renew_subs 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14580b7a40 con 0x7f14581430c0 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 10 monclient: _renew_subs 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581360e0 con 0x7f14581430c0 2024-03-25T18:26:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.472+0000 7f145d38c640 1 librados: init done 2024-03-25T18:26:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.473+0000 7f145d38c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.473+0000 7f145d38c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.473+0000 7f145d38c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.473+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1458135430 con 0x7f14581430c0 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1450002060 con 0x7f14581430c0 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1450003b10 con 0x7f14581430c0 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f143f7fe640 10 monclient: dump: 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:12.775 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:12.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.774+0000 7f14557fa640 4 set_mon_vals no callback set 2024-03-25T18:26:12.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.775+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f1450003d30 con 0x7f14581430c0 2024-03-25T18:26:12.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.775+0000 7f1456ffd640 10 monclient: get_auth_request con 0x7f144406df70 auth_method 0 2024-03-25T18:26:12.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.775+0000 7f143f7fe640 10 client.4313.objecter ms_handle_connect 0x7f144406df70 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f1450053850 con 0x7f14581430c0 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 10 client.4313.objecter ms_dispatch 0x7f1458000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f145004f360 con 0x7f14581430c0 2024-03-25T18:26:12.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:12.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:12.780+0000 7f143f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:13.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.020+0000 7f145d38c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:13.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.020+0000 7f145d38c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:13.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.020+0000 7f145d38c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:13.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.020+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14580627b0 con 0x7f14581430c0 2024-03-25T18:26:13.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.056+0000 7f143f7fe640 1 -- 172.21.15.176:0/267221681 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f1450051290 con 0x7f14581430c0 2024-03-25T18:26:13.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.056+0000 7f143f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:13.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.056+0000 7f143f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:13.058 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:13.058 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 10 librados: watch_flush enter 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 10 librados: watch_flush exit 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 >> v1:172.21.15.176:6800/2598213983 conn(0x7f144406df70 legacy=0x7f1444070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 10 monclient: shutdown 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:13.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 >> v1:172.21.15.176:6790/0 conn(0x7f14581430c0 legacy=0x7f14581d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 shutdown_connections 2024-03-25T18:26:13.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 >> 172.21.15.176:0/267221681 conn(0x7f14580b7ea0 msgr2=0x7f1458074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:13.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 shutdown_connections 2024-03-25T18:26:13.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 -- 172.21.15.176:0/267221681 wait complete. 2024-03-25T18:26:13.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:13.060+0000 7f145d38c640 1 librados: shutdown 2024-03-25T18:26:13.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:19.880 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:19.993 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.992+0000 7fce1a710640 10 monclient: get_monmap_and_config 2024-03-25T18:26:19.993 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.992+0000 7fce1a710640 10 monclient: build_initial_monmap 2024-03-25T18:26:19.993 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.992+0000 7fce1a710640 10 monclient: monmap: 2024-03-25T18:26:19.993 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:19.993 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:19.993558+0000 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:19.993558+0000 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:19.994 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.993+0000 7fce1a710640 1 Processor -- start 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.993+0000 7fce1a710640 1 -- start start 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.993+0000 7fce1a710640 10 monclient: init 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:19.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-c con 0x7fce140a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-b con 0x7fce1413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-a con 0x7fce141430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce14142520 con 0x7fce141430c0 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce140b28a0 con 0x7fce140a98f0 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce140b2a30 con 0x7fce1413f710 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:19.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:19.994+0000 7fce1a710640 10 monclient(hunting): authenticate will time out at 8258.306768s 2024-03-25T18:26:20.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.005+0000 7fce137fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fce1413f710 0x7fce141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53756/0 (socket says 172.21.15.176:53756) 2024-03-25T18:26:20.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.005+0000 7fce137fe640 1 -- 172.21.15.176:0/3893076540 learned_addr learned my addr 172.21.15.176:0/3893076540 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:20.022 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056149309 0 0) 0x7fce14142520 con 0x7fce141430c0 2024-03-25T18:26:20.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:20.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:20.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:20.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:26:20.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.021+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc003520 con 0x7fce141430c0 2024-03-25T18:26:20.028 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719759261 0 0) 0x7fcdfc003520 con 0x7fce141430c0 2024-03-25T18:26:20.028 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 >> v1:172.21.15.176:6790/0 conn(0x7fce1413f710 legacy=0x7fce141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.028 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 >> v1:172.21.15.186:6789/0 conn(0x7fce140a98f0 legacy=0x7fce140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 monclient: found mon.noname-a 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce140b44c0 con 0x7fce141430c0 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fce0c001a10 con 0x7fce141430c0 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce1a710640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce1a710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:20.029 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce12ffd640 10 monclient: dump: 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.027+0000 7fce1a710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:20.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce0c001b70 con 0x7fce141430c0 2024-03-25T18:26:20.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:20.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 1 -- 172.21.15.176:0/3893076540 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fce0c001f80 con 0x7fce141430c0 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce12ffd640 10 monclient: dump: 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:20.039 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:20.040 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:20.040 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:20.040 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce1a710640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:20.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.037+0000 7fce1a710640 4 set_mon_vals no callback set 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 10 monclient: shutdown 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 1 -- 172.21.15.176:0/3893076540 >> v1:172.21.15.176:6789/0 conn(0x7fce141430c0 legacy=0x7fce1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 1 -- 172.21.15.176:0/3893076540 shutdown_connections 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 1 -- 172.21.15.176:0/3893076540 >> 172.21.15.176:0/3893076540 conn(0x7fce140b7ea0 msgr2=0x7fce140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 1 -- 172.21.15.176:0/3893076540 shutdown_connections 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.038+0000 7fce1a710640 1 -- 172.21.15.176:0/3893076540 wait complete. 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.039+0000 7fce1a710640 10 monclient: build_initial_monmap 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.039+0000 7fce1a710640 10 monclient: monmap: 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:20.044 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.039+0000 7fce1a710640 1 librados: starting msgr at 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.039+0000 7fce1a710640 1 librados: starting objecter 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.039+0000 7fce1a710640 1 Processor -- start 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.040+0000 7fce1a710640 1 -- start start 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.040+0000 7fce1a710640 1 librados: setting wanted keys 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.040+0000 7fce1a710640 1 librados: calling monclient init 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.040+0000 7fce1a710640 10 monclient: init 2024-03-25T18:26:20.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-a con 0x7fce140a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-b con 0x7fce1413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient(hunting): picked mon.noname-c con 0x7fce141430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce14070ff0 con 0x7fce140a98f0 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1406ff00 con 0x7fce1413f710 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce140701e0 con 0x7fce141430c0 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:20.046 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.041+0000 7fce1a710640 10 monclient(hunting): authenticate will time out at 8258.353042s 2024-03-25T18:26:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.050+0000 7fce13fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fce140a98f0 0x7fce140708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:50958/0 (socket says 172.21.15.176:50958) 2024-03-25T18:26:20.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.050+0000 7fce13fff640 1 -- 172.21.15.176:0/554947888 learned_addr learned my addr 172.21.15.176:0/554947888 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:20.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.053+0000 7fce10ff9640 10 client.?.objecter ms_handle_connect 0x7fce141430c0 2024-03-25T18:26:20.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.053+0000 7fce10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:20.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.053+0000 7fce10ff9640 10 client.?.objecter ms_handle_connect 0x7fce1413f710 2024-03-25T18:26:20.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.053+0000 7fce10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:20.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.055+0000 7fce10ff9640 10 client.?.objecter ms_handle_connect 0x7fce140a98f0 2024-03-25T18:26:20.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.055+0000 7fce10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:20.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176901895 0 0) 0x7fce140701e0 con 0x7fce141430c0 2024-03-25T18:26:20.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:20.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:20.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.069+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce00004eb0 con 0x7fce141430c0 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581117391 0 0) 0x7fce1406ff00 con 0x7fce1413f710 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:20.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:26:20.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.070+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce140701e0 con 0x7fce1413f710 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530421324 0 0) 0x7fce14070ff0 con 0x7fce140a98f0 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:26:20.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.072+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1406ff00 con 0x7fce140a98f0 2024-03-25T18:26:20.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.081+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983426205 0 0) 0x7fce1406ff00 con 0x7fce140a98f0 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce14070ff0 con 0x7fce140a98f0 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fce040018f0 con 0x7fce140a98f0 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:20.083 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:20.084 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:20.084 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.082+0000 7fce10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 49578467 0 0) 0x7fce00004eb0 con 0x7fce141430c0 2024-03-25T18:26:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce1406ff00 con 0x7fce141430c0 2024-03-25T18:26:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fce0c001f40 con 0x7fce141430c0 2024-03-25T18:26:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.091+0000 7fce10ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3627330290 0 0) 0x7fce14070ff0 con 0x7fce140a98f0 2024-03-25T18:26:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 >> v1:172.21.15.176:6790/0 conn(0x7fce141430c0 legacy=0x7fce141e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 >> v1:172.21.15.186:6789/0 conn(0x7fce1413f710 legacy=0x7fce1406e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 1 monclient: found mon.a 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce140704c0 con 0x7fce140a98f0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 10 monclient: _renew_subs 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1406f200 con 0x7fce140a98f0 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:20.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 10 monclient: _renew_subs 2024-03-25T18:26:20.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1414d2d0 con 0x7fce140a98f0 2024-03-25T18:26:20.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.092+0000 7fce1a710640 1 librados: init done 2024-03-25T18:26:20.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.093+0000 7fce1a710640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:20.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.093+0000 7fce1a710640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:20.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.093+0000 7fce1a710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.093+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1414d5d0 con 0x7fce140a98f0 2024-03-25T18:26:20.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce04003610 con 0x7fce140a98f0 2024-03-25T18:26:20.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fce04003960 con 0x7fce140a98f0 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce10ff9640 10 monclient: dump: 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:20.398 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:20.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.396+0000 7fce127fc640 4 set_mon_vals no callback set 2024-03-25T18:26:20.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.397+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fce04003b80 con 0x7fce140a98f0 2024-03-25T18:26:20.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.398+0000 7fce137fe640 10 monclient: get_auth_request con 0x7fce0006df70 auth_method 0 2024-03-25T18:26:20.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.398+0000 7fce10ff9640 10 client.4302.objecter ms_handle_connect 0x7fce0006df70 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fce04054770 con 0x7fce140a98f0 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 10 client.4302.objecter ms_dispatch 0x7fce14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce04050280 con 0x7fce140a98f0 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:20.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.402+0000 7fce10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:20.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.673+0000 7fce1a710640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.673+0000 7fce1a710640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.673+0000 7fce1a710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:20.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.673+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce140627b0 con 0x7fce140a98f0 2024-03-25T18:26:20.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.702+0000 7fce10ff9640 1 -- 172.21.15.176:0/554947888 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fce040521b0 con 0x7fce140a98f0 2024-03-25T18:26:20.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.702+0000 7fce10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:20.703 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.702+0000 7fce10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:20.704 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:20.704 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:20.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.704+0000 7fce1a710640 10 librados: watch_flush enter 2024-03-25T18:26:20.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 10 librados: watch_flush exit 2024-03-25T18:26:20.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:26:20.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:26:20.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 >> v1:172.21.15.176:6800/2598213983 conn(0x7fce0006df70 legacy=0x7fce00070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 10 monclient: shutdown 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 >> v1:172.21.15.176:6789/0 conn(0x7fce140a98f0 legacy=0x7fce140708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 shutdown_connections 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 >> 172.21.15.176:0/554947888 conn(0x7fce140b7ea0 msgr2=0x7fce14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 shutdown_connections 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 -- 172.21.15.176:0/554947888 wait complete. 2024-03-25T18:26:20.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:20.705+0000 7fce1a710640 1 librados: shutdown 2024-03-25T18:26:20.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:27.528 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:27.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.642+0000 7fdfb4944640 10 monclient: get_monmap_and_config 2024-03-25T18:26:27.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.642+0000 7fdfb4944640 10 monclient: build_initial_monmap 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.642+0000 7fdfb4944640 10 monclient: monmap: 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:27.643627+0000 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:27.643627+0000 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:27.644 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.643+0000 7fdfb4944640 1 Processor -- start 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.643+0000 7fdfb4944640 1 -- start start 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.643+0000 7fdfb4944640 10 monclient: init 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb00b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb00b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00af940 con 0x7fdfb00b4d60 2024-03-25T18:26:27.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00b0a70 con 0x7fdfb014be90 2024-03-25T18:26:27.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00b0d50 con 0x7fdfb00b1730 2024-03-25T18:26:27.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:27.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.644+0000 7fdfb4944640 10 monclient(hunting): authenticate will time out at 8265.956297s 2024-03-25T18:26:27.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.646+0000 7fdfaed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdfb014be90 0x7fdfb014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:53474/0 (socket says 172.21.15.176:53474) 2024-03-25T18:26:27.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.646+0000 7fdfaed9d640 1 -- 172.21.15.176:0/3950740692 learned_addr learned my addr 172.21.15.176:0/3950740692 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:27.706 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:29.463 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.719 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:29.475 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.754 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:29.511 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.760 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:29.515 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.777 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:29.509 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.778 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:29.531 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460737867 0 0) 0x7fdfb00b0a70 con 0x7fdfb014be90 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:26:27.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.777+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb014be90 2024-03-25T18:26:27.780 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:29.514 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212700982 0 0) 0x7fdfb00b0d50 con 0x7fdfb00b1730 2024-03-25T18:26:27.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:26:27.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.783+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb00b0a70 con 0x7fdfb00b1730 2024-03-25T18:26:27.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1185705009 0 0) 0x7fdf9c003520 con 0x7fdfb014be90 2024-03-25T18:26:27.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 >> v1:172.21.15.176:6790/0 conn(0x7fdfb00b1730 legacy=0x7fdfb00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 >> v1:172.21.15.176:6789/0 conn(0x7fdfb00b4d60 legacy=0x7fdfb013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 monclient: found mon.noname-c 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00b1030 con 0x7fdfb014be90 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdfa8001a10 con 0x7fdfb014be90 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfb4944640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfb4944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:27.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfadd9b640 10 monclient: dump: 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:27.796 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:27.797 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:27.797 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:27.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.793+0000 7fdfb4944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:27.809 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:29.565 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.813 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:29.547 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfa8001b70 con 0x7fdfb014be90 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 1 -- 172.21.15.176:0/3950740692 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdfa8001f80 con 0x7fdfb014be90 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfb4944640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfb4944640 4 set_mon_vals no callback set 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.849+0000 7fdfadd9b640 10 monclient: dump: 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:27.850 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:27.851 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:27.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 10 monclient: shutdown 2024-03-25T18:26:27.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:27.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 1 -- 172.21.15.176:0/3950740692 >> v1:172.21.15.186:6789/0 conn(0x7fdfb014be90 legacy=0x7fdfb014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 1 -- 172.21.15.176:0/3950740692 shutdown_connections 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 1 -- 172.21.15.176:0/3950740692 >> 172.21.15.176:0/3950740692 conn(0x7fdfb00617a0 msgr2=0x7fdfb0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 1 -- 172.21.15.176:0/3950740692 shutdown_connections 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.850+0000 7fdfb4944640 1 -- 172.21.15.176:0/3950740692 wait complete. 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.851+0000 7fdfb4944640 10 monclient: build_initial_monmap 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.851+0000 7fdfb4944640 10 monclient: monmap: 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:27.853 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.851+0000 7fdfb4944640 1 librados: starting msgr at 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.851+0000 7fdfb4944640 1 librados: starting objecter 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.851+0000 7fdfb4944640 1 Processor -- start 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 -- start start 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 librados: setting wanted keys 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 librados: calling monclient init 2024-03-25T18:26:27.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient: init 2024-03-25T18:26:27.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:27.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:27.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb00b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:27.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb00b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:27.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:27.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb01d75d0 con 0x7fdfb014be90 2024-03-25T18:26:27.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0141e10 con 0x7fdfb00b4d60 2024-03-25T18:26:27.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0142080 con 0x7fdfb00b1730 2024-03-25T18:26:27.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:27.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.852+0000 7fdfb4944640 10 monclient(hunting): authenticate will time out at 8266.164707s 2024-03-25T18:26:27.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.865+0000 7fdfaed9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdfb014be90 0x7fdfb01daf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49734/0 (socket says 172.21.15.176:49734) 2024-03-25T18:26:27.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.865+0000 7fdfaed9d640 1 -- 172.21.15.176:0/474320225 learned_addr learned my addr 172.21.15.176:0/474320225 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:27.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.866+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb014be90 2024-03-25T18:26:27.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.866+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.871+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb00b1730 2024-03-25T18:26:27.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.871+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269554248 0 0) 0x7fdfb01d75d0 con 0x7fdfb014be90 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:26:27.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.882+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf90004eb0 con 0x7fdfb014be90 2024-03-25T18:26:27.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.883+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb00b4d60 2024-03-25T18:26:27.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.883+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559390468 0 0) 0x7fdfb0142080 con 0x7fdfb00b1730 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:26:27.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.898+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb01d75d0 con 0x7fdfb00b1730 2024-03-25T18:26:27.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294707502 0 0) 0x7fdf90004eb0 con 0x7fdfb014be90 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb0142080 con 0x7fdfb014be90 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdfa8001f40 con 0x7fdfb014be90 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:27.903 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:27.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.902+0000 7fdfa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:27.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3286541664 0 0) 0x7fdfb0142080 con 0x7fdfb014be90 2024-03-25T18:26:27.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 >> v1:172.21.15.176:6790/0 conn(0x7fdfb00b1730 legacy=0x7fdfb01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 >> v1:172.21.15.186:6789/0 conn(0x7fdfb00b4d60 legacy=0x7fdfb01d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 1 monclient: found mon.a 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0140d00 con 0x7fdfb014be90 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 5 monclient: authenticate success, global_id 4308 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 10 monclient: _renew_subs 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb0135a90 con 0x7fdfb014be90 2024-03-25T18:26:27.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 10 monclient: _renew_subs 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb01d83a0 con 0x7fdfb014be90 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.918+0000 7fdfb4944640 1 librados: init done 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.919+0000 7fdfb4944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.919+0000 7fdfb4944640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.919+0000 7fdfb4944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:27.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:27.919+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb00627b0 con 0x7fdfb014be90 2024-03-25T18:26:28.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.225+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfa8004e10 con 0x7fdfb014be90 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.225+0000 7fdfa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.225+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdfa8005120 con 0x7fdfb014be90 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.225+0000 7fdfa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.226+0000 7fdfa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.226+0000 7fdfa6ffd640 10 monclient: dump: 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:28.227 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.226+0000 7fdfad59a640 4 set_mon_vals no callback set 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.226+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fdfa801b0f0 con 0x7fdfb014be90 2024-03-25T18:26:28.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.227+0000 7fdfae59c640 10 monclient: get_auth_request con 0x7fdf9006dd10 auth_method 0 2024-03-25T18:26:28.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.227+0000 7fdfa6ffd640 10 client.4308.objecter ms_handle_connect 0x7fdf9006dd10 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fdfa8054b60 con 0x7fdfb014be90 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 10 client.4308.objecter ms_dispatch 0x7fdfb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfa80504a0 con 0x7fdfb014be90 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:28.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.233+0000 7fdfa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:28.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.449+0000 7fdfb4944640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.449+0000 7fdfb4944640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.449+0000 7fdfb4944640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:26:28.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.449+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfb00b22c0 con 0x7fdfb014be90 2024-03-25T18:26:28.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.506+0000 7fdfa6ffd640 1 -- 172.21.15.176:0/474320225 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fdfa80525a0 con 0x7fdfb014be90 2024-03-25T18:26:28.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.506+0000 7fdfa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:28.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.506+0000 7fdfa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:28.508 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:28.508 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 10 librados: watch_flush enter 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 10 librados: watch_flush exit 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 10 client.4308.objecter successfully canceled tick 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 >> v1:172.21.15.176:6800/2598213983 conn(0x7fdf9006dd10 legacy=0x7fdf900701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 10 monclient: shutdown 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 >> v1:172.21.15.176:6789/0 conn(0x7fdfb014be90 legacy=0x7fdfb01daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 shutdown_connections 2024-03-25T18:26:28.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.509+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 >> 172.21.15.176:0/474320225 conn(0x7fdfb00617a0 msgr2=0x7fdfb00b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:28.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.510+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 shutdown_connections 2024-03-25T18:26:28.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.510+0000 7fdfb4944640 1 -- 172.21.15.176:0/474320225 wait complete. 2024-03-25T18:26:28.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:28.510+0000 7fdfb4944640 1 librados: shutdown 2024-03-25T18:26:28.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:35.329 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:35.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.441+0000 7f227b0e2640 10 monclient: get_monmap_and_config 2024-03-25T18:26:35.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.441+0000 7f227b0e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.441+0000 7f227b0e2640 10 monclient: monmap: 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:35.443569+0000 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:35.443569+0000 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:35.444 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.442+0000 7f227b0e2640 1 Processor -- start 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.442+0000 7f227b0e2640 1 -- start start 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.442+0000 7f227b0e2640 10 monclient: init 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f22740a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f227413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f22741430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2274142520 con 0x7f22740a98f0 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22740b28a0 con 0x7f22741430c0 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22740b2a30 con 0x7f227413f710 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:35.446 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.443+0000 7f227b0e2640 10 monclient(hunting): authenticate will time out at 8273.756679s 2024-03-25T18:26:35.447 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.446+0000 7f2273fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f227413f710 0x7f22741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:59880/0 (socket says 172.21.15.176:59880) 2024-03-25T18:26:35.447 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.446+0000 7f2273fff640 1 -- 172.21.15.176:0/2337322418 learned_addr learned my addr 172.21.15.176:0/2337322418 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:35.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933643644 0 0) 0x7f22740b2a30 con 0x7f227413f710 2024-03-25T18:26:35.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:26:35.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.462+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2264003520 con 0x7f227413f710 2024-03-25T18:26:35.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602103398 0 0) 0x7f22740b28a0 con 0x7f22741430c0 2024-03-25T18:26:35.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:26:35.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.471+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22740b2a30 con 0x7f22741430c0 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481817544 0 0) 0x7f2274142520 con 0x7f22740a98f0 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:26:35.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.483+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22740b28a0 con 0x7f22740a98f0 2024-03-25T18:26:35.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784563607 0 0) 0x7f2264003520 con 0x7f227413f710 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 >> v1:172.21.15.186:6789/0 conn(0x7f22741430c0 legacy=0x7f227414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 >> v1:172.21.15.176:6789/0 conn(0x7f22740a98f0 legacy=0x7f22740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 monclient: found mon.noname-b 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22740b44c0 con 0x7f227413f710 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2260001690 con 0x7f227413f710 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f227b0e2640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f227b0e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:35.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f22737fe640 10 monclient: dump: 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.484+0000 7f227b0e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:35.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22600018b0 con 0x7f227413f710 2024-03-25T18:26:35.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 1 -- 172.21.15.176:0/2337322418 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2260001c40 con 0x7f227413f710 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f22737fe640 10 monclient: dump: 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f227b0e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:35.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.512+0000 7f227b0e2640 4 set_mon_vals no callback set 2024-03-25T18:26:35.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 10 monclient: shutdown 2024-03-25T18:26:35.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:35.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 1 -- 172.21.15.176:0/2337322418 >> v1:172.21.15.176:6790/0 conn(0x7f227413f710 legacy=0x7f22741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 1 -- 172.21.15.176:0/2337322418 shutdown_connections 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 1 -- 172.21.15.176:0/2337322418 >> 172.21.15.176:0/2337322418 conn(0x7f22740b7ea0 msgr2=0x7f22740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 1 -- 172.21.15.176:0/2337322418 shutdown_connections 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.513+0000 7f227b0e2640 1 -- 172.21.15.176:0/2337322418 wait complete. 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.514+0000 7f227b0e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.514+0000 7f227b0e2640 10 monclient: monmap: 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:35.519 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 librados: starting msgr at 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 librados: starting objecter 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 Processor -- start 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 -- start start 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 librados: setting wanted keys 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 1 librados: calling monclient init 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient: init 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f22740a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f227413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.515+0000 7f227b0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f22741430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.516+0000 7f227b0e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22741d7490 con 0x7f22740a98f0 2024-03-25T18:26:35.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.516+0000 7f227b0e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22741d76c0 con 0x7f227413f710 2024-03-25T18:26:35.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.516+0000 7f227b0e2640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22741d82f0 con 0x7f22741430c0 2024-03-25T18:26:35.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.516+0000 7f227b0e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:35.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.516+0000 7f227b0e2640 10 monclient(hunting): authenticate will time out at 8273.828918s 2024-03-25T18:26:35.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.526+0000 7f2273fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f227413f710 0x7f22741383e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:42126/0 (socket says 172.21.15.176:42126) 2024-03-25T18:26:35.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.526+0000 7f2273fff640 1 -- 172.21.15.176:0/891002976 learned_addr learned my addr 172.21.15.176:0/891002976 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:35.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.528+0000 7f22717fa640 10 client.?.objecter ms_handle_connect 0x7f227413f710 2024-03-25T18:26:35.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.528+0000 7f22717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.533+0000 7f22717fa640 10 client.?.objecter ms_handle_connect 0x7f22741430c0 2024-03-25T18:26:35.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.533+0000 7f22717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.534+0000 7f22717fa640 10 client.?.objecter ms_handle_connect 0x7f22740a98f0 2024-03-25T18:26:35.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.534+0000 7f22717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607449361 0 0) 0x7f22741d76c0 con 0x7f227413f710 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:26:35.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.535+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2258004eb0 con 0x7f227413f710 2024-03-25T18:26:35.540 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777113040 0 0) 0x7f22741d82f0 con 0x7f22741430c0 2024-03-25T18:26:35.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:26:35.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.539+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22741d76c0 con 0x7f22741430c0 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994741834 0 0) 0x7f22741d7490 con 0x7f22740a98f0 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:26:35.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.545+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22741d82f0 con 0x7f22740a98f0 2024-03-25T18:26:35.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3497391996 0 0) 0x7f2258004eb0 con 0x7f227413f710 2024-03-25T18:26:35.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22741d7490 con 0x7f227413f710 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2260001690 con 0x7f227413f710 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 10 monclient(hunting): dump: 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:35.550 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.551 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.547+0000 7f22717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 177224580 0 0) 0x7f22741d76c0 con 0x7f22741430c0 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2258004eb0 con 0x7f22741430c0 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f226c001a90 con 0x7f22741430c0 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:35.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.554+0000 7f22717fa640 10 monclient(hunting): dump: 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:35.557 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473989937 0 0) 0x7f2258004eb0 con 0x7f22741430c0 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 >> v1:172.21.15.186:6789/0 conn(0x7f227413f710 legacy=0x7f22741383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 >> v1:172.21.15.176:6789/0 conn(0x7f22740a98f0 legacy=0x7f22741d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 1 monclient: found mon.c 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2274138af0 con 0x7f22741430c0 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:35.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f22717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 10 monclient: _renew_subs 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22741376d0 con 0x7f22741430c0 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 10 monclient: _renew_subs 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2274134fc0 con 0x7f22741430c0 2024-03-25T18:26:35.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.561+0000 7f227b0e2640 1 librados: init done 2024-03-25T18:26:35.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.563+0000 7f227b0e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.563+0000 7f227b0e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.563+0000 7f227b0e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:35.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.563+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22741d7850 con 0x7f22741430c0 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f226c001f80 con 0x7f22741430c0 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f226c003b30 con 0x7f22741430c0 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f22717fa640 10 monclient: dump: 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:35.856 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.853+0000 7f2272ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.854+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f226c019b00 con 0x7f22741430c0 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.855+0000 7f2278e7e640 10 monclient: get_auth_request con 0x7f225806dfb0 auth_method 0 2024-03-25T18:26:35.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.855+0000 7f22717fa640 10 client.4331.objecter ms_handle_connect 0x7f225806dfb0 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f226c053800 con 0x7f22741430c0 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 10 client.4331.objecter ms_dispatch 0x7f2274000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f226c04f140 con 0x7f22741430c0 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:35.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:35.860+0000 7f22717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:36.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.086+0000 7f227b0e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:36.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.086+0000 7f227b0e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:36.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.086+0000 7f227b0e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:36.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.086+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22740627b0 con 0x7f22741430c0 2024-03-25T18:26:36.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.115+0000 7f22717fa640 1 -- 172.21.15.176:0/891002976 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f226c051240 con 0x7f22741430c0 2024-03-25T18:26:36.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.115+0000 7f22717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:36.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.115+0000 7f22717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:36.118 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:36.118 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:36.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 10 librados: watch_flush enter 2024-03-25T18:26:36.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 10 librados: watch_flush exit 2024-03-25T18:26:36.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-25T18:26:36.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 10 client.4331.objecter successfully canceled tick 2024-03-25T18:26:36.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 >> v1:172.21.15.176:6800/2598213983 conn(0x7f225806dfb0 legacy=0x7f2258070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 10 monclient: shutdown 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 >> v1:172.21.15.176:6790/0 conn(0x7f22741430c0 legacy=0x7f2274133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 shutdown_connections 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 >> 172.21.15.176:0/891002976 conn(0x7f22740b7ea0 msgr2=0x7f2274074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 shutdown_connections 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 -- 172.21.15.176:0/891002976 wait complete. 2024-03-25T18:26:36.121 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:36.118+0000 7f227b0e2640 1 librados: shutdown 2024-03-25T18:26:36.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:42.940 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:43.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 10 monclient: get_monmap_and_config 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 10 monclient: monmap: 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:43.050975+0000 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:43.050975+0000 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:43.051 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 1 Processor -- start 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 1 -- start start 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.049+0000 7fc65f315640 10 monclient: init 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-a con 0x7fc6580a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-b con 0x7fc65813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-c con 0x7fc6581430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc658142520 con 0x7fc6580a98f0 2024-03-25T18:26:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580b28a0 con 0x7fc6581430c0 2024-03-25T18:26:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580b2a30 con 0x7fc65813f710 2024-03-25T18:26:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.050+0000 7fc65f315640 10 monclient(hunting): authenticate will time out at 8281.363628s 2024-03-25T18:26:43.060 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.059+0000 7fc65d8b2640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc6581430c0 0x7fc65814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:46864/0 (socket says 172.21.15.176:46864) 2024-03-25T18:26:43.060 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.059+0000 7fc65d8b2640 1 -- 172.21.15.176:0/1118785892 learned_addr learned my addr 172.21.15.176:0/1118785892 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:43.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332168291 0 0) 0x7fc6580b28a0 con 0x7fc6581430c0 2024-03-25T18:26:43.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:26:43.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.079+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc644003520 con 0x7fc6581430c0 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921573369 0 0) 0x7fc658142520 con 0x7fc6580a98f0 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:26:43.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.083+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6580b28a0 con 0x7fc6580a98f0 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519554085 0 0) 0x7fc6580b2a30 con 0x7fc65813f710 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:26:43.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.087+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658142520 con 0x7fc65813f710 2024-03-25T18:26:43.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322232729 0 0) 0x7fc644003520 con 0x7fc6581430c0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 >> v1:172.21.15.176:6790/0 conn(0x7fc65813f710 legacy=0x7fc6581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 >> v1:172.21.15.176:6789/0 conn(0x7fc6580a98f0 legacy=0x7fc6580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 monclient: found mon.noname-c 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580b44c0 con 0x7fc6581430c0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc650001a10 con 0x7fc6581430c0 2024-03-25T18:26:43.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc64ffff640 10 monclient: dump: 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc65f315640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:26:43.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.099+0000 7fc65f315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:43.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc650001b70 con 0x7fc6581430c0 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 1 -- 172.21.15.176:0/1118785892 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc650001f80 con 0x7fc6581430c0 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.122+0000 7fc64ffff640 10 monclient: dump: 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:43.125 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:43.126 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:43.126 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:43.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 4 set_mon_vals no callback set 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 10 monclient: shutdown 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 1 -- 172.21.15.176:0/1118785892 >> v1:172.21.15.186:6789/0 conn(0x7fc6581430c0 legacy=0x7fc65814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 1 -- 172.21.15.176:0/1118785892 shutdown_connections 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.123+0000 7fc65f315640 1 -- 172.21.15.176:0/1118785892 >> 172.21.15.176:0/1118785892 conn(0x7fc6580b7ea0 msgr2=0x7fc6580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.124+0000 7fc65f315640 1 -- 172.21.15.176:0/1118785892 shutdown_connections 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.124+0000 7fc65f315640 1 -- 172.21.15.176:0/1118785892 wait complete. 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.124+0000 7fc65f315640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.124+0000 7fc65f315640 10 monclient: monmap: 2024-03-25T18:26:43.128 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 librados: starting msgr at 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 librados: starting objecter 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 Processor -- start 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 -- start start 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 librados: setting wanted keys 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 1 librados: calling monclient init 2024-03-25T18:26:43.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 10 monclient: init 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.125+0000 7fc65f315640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-a con 0x7fc6580a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-b con 0x7fc65813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient(hunting): picked mon.noname-c con 0x7fc6581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6581369d0 con 0x7fc6580a98f0 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc658133a60 con 0x7fc65813f710 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580ab9e0 con 0x7fc6581430c0 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.126+0000 7fc65f315640 10 monclient(hunting): authenticate will time out at 8281.439062s 2024-03-25T18:26:43.136 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.133+0000 7fc65d8b2640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc6581430c0 0x7fc658133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:36546/0 (socket says 172.21.15.176:36546) 2024-03-25T18:26:43.136 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.133+0000 7fc65d8b2640 1 -- 172.21.15.176:0/220504472 learned_addr learned my addr 172.21.15.176:0/220504472 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:43.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.136+0000 7fc64dffb640 10 client.?.objecter ms_handle_connect 0x7fc6581430c0 2024-03-25T18:26:43.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.136+0000 7fc64dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.138+0000 7fc64dffb640 10 client.?.objecter ms_handle_connect 0x7fc65813f710 2024-03-25T18:26:43.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.138+0000 7fc64dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.138+0000 7fc64dffb640 10 client.?.objecter ms_handle_connect 0x7fc6580a98f0 2024-03-25T18:26:43.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.138+0000 7fc64dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933179988 0 0) 0x7fc6580ab9e0 con 0x7fc6581430c0 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:26:43.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.142+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c004eb0 con 0x7fc6581430c0 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736614782 0 0) 0x7fc6581369d0 con 0x7fc6580a98f0 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:26:43.149 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.147+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6580ab9e0 con 0x7fc6580a98f0 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728638398 0 0) 0x7fc63c004eb0 con 0x7fc6581430c0 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6581369d0 con 0x7fc6581430c0 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc650001ec0 con 0x7fc6581430c0 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:43.157 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394583927 0 0) 0x7fc658133a60 con 0x7fc65813f710 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:26:43.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.155+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c004eb0 con 0x7fc65813f710 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 548804683 0 0) 0x7fc6580ab9e0 con 0x7fc6580a98f0 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc658133a60 con 0x7fc6580a98f0 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc654001970 con 0x7fc6580a98f0 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:43.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.158+0000 7fc64dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:43.161 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4097159636 0 0) 0x7fc6581369d0 con 0x7fc6581430c0 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 >> v1:172.21.15.186:6789/0 conn(0x7fc65813f710 legacy=0x7fc6581362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 >> v1:172.21.15.176:6789/0 conn(0x7fc6580a98f0 legacy=0x7fc658132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 1 monclient: found mon.c 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580abcc0 con 0x7fc6581430c0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc64dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc65f315640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc65f315640 10 monclient: _renew_subs 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc65f315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.161+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6580abfa0 con 0x7fc6581430c0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.162+0000 7fc65f315640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.162+0000 7fc65f315640 10 monclient: _renew_subs 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.162+0000 7fc65f315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.162+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc658137e40 con 0x7fc6581430c0 2024-03-25T18:26:43.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.162+0000 7fc65f315640 1 librados: init done 2024-03-25T18:26:43.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.163+0000 7fc65f315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.163+0000 7fc65f315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.163+0000 7fc65f315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:43.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.163+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6580627b0 con 0x7fc6581430c0 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc650004e10 con 0x7fc6581430c0 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6500015d0 con 0x7fc6581430c0 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:43.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 10 monclient: dump: 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:43.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64f7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:43.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.476+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fc65001af90 con 0x7fc6581430c0 2024-03-25T18:26:43.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.477+0000 7fc65d0b1640 10 monclient: get_auth_request con 0x7fc63c06df20 auth_method 0 2024-03-25T18:26:43.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.477+0000 7fc64dffb640 10 client.4337.objecter ms_handle_connect 0x7fc63c06df20 2024-03-25T18:26:43.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fc650054b90 con 0x7fc6581430c0 2024-03-25T18:26:43.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 10 client.4337.objecter ms_dispatch 0x7fc658000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:43.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:43.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:43.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T18:26:43.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6500504d0 con 0x7fc6581430c0 2024-03-25T18:26:43.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.485 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.482+0000 7fc64dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:43.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.711+0000 7fc65f315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.711+0000 7fc65f315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.711+0000 7fc65f315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:43.714 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.711+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc65813dfd0 con 0x7fc6581430c0 2024-03-25T18:26:43.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.745+0000 7fc64dffb640 1 -- 172.21.15.176:0/220504472 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fc6500525d0 con 0x7fc6581430c0 2024-03-25T18:26:43.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.745+0000 7fc64dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:43.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.745+0000 7fc64dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:43.747 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:43.747 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:43.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 10 librados: watch_flush enter 2024-03-25T18:26:43.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 10 librados: watch_flush exit 2024-03-25T18:26:43.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 10 client.4337.objecter successfully canceled tick 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 >> v1:172.21.15.176:6800/2598213983 conn(0x7fc63c06df20 legacy=0x7fc63c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 10 monclient: shutdown 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 >> v1:172.21.15.176:6790/0 conn(0x7fc6581430c0 legacy=0x7fc658133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 shutdown_connections 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 >> 172.21.15.176:0/220504472 conn(0x7fc6580b7ea0 msgr2=0x7fc658074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 shutdown_connections 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 -- 172.21.15.176:0/220504472 wait complete. 2024-03-25T18:26:43.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:43.748+0000 7fc65f315640 1 librados: shutdown 2024-03-25T18:26:43.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:47.724 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:49.480 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.788 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:49.544 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.821 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:02:49.577 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.832 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:49.566 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.845 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:49.602 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.850 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:49.585 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.859 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:49.615 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.872 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:49.606 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.887 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:02:49.644 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:50.569 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.684+0000 7f56e1379640 10 monclient: get_monmap_and_config 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.684+0000 7f56e1379640 10 monclient: build_initial_monmap 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.684+0000 7f56e1379640 10 monclient: monmap: 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:50.686188+0000 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:50.686188+0000 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:50.686 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:50.687 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:50.687 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:50.687 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:50.687 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.684+0000 7f56e1379640 1 Processor -- start 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 1 -- start start 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient: init 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-c con 0x7f56dc0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-a con 0x7f56dc0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:50.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-b con 0x7f56dc14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:50.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0af940 con 0x7f56dc0b1730 2024-03-25T18:26:50.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0b0a70 con 0x7f56dc0b4d60 2024-03-25T18:26:50.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0b0d50 con 0x7f56dc14be90 2024-03-25T18:26:50.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:50.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.685+0000 7f56e1379640 10 monclient(hunting): authenticate will time out at 8288.998635s 2024-03-25T18:26:50.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.690+0000 7f56da7fc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f56dc0b1730 0x7f56dc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:42198/0 (socket says 172.21.15.176:42198) 2024-03-25T18:26:50.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.690+0000 7f56da7fc640 1 -- 172.21.15.176:0/710763929 learned_addr learned my addr 172.21.15.176:0/710763929 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994502717 0 0) 0x7f56dc0af940 con 0x7f56dc0b1730 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:26:50.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.702+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c4003520 con 0x7f56dc0b1730 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142868118 0 0) 0x7f56dc0b0d50 con 0x7f56dc14be90 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:26:50.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.716+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc0af940 con 0x7f56dc14be90 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952422757 0 0) 0x7f56dc0b0a70 con 0x7f56dc0b4d60 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc0b0d50 con 0x7f56dc0b4d60 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800732768 0 0) 0x7f56dc0af940 con 0x7f56dc14be90 2024-03-25T18:26:50.726 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 >> v1:172.21.15.186:6789/0 conn(0x7f56dc0b4d60 legacy=0x7f56dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 >> v1:172.21.15.176:6789/0 conn(0x7f56dc0b1730 legacy=0x7f56dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 monclient: found mon.noname-b 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0b1030 con 0x7f56dc14be90 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f56d40019f0 con 0x7f56dc14be90 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56d9ffb640 10 monclient: dump: 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:50.727 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56e1379640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:26:50.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.723+0000 7f56e1379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56d4001b50 con 0x7f56dc14be90 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 1 -- 172.21.15.176:0/710763929 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f56d4001f60 con 0x7f56dc14be90 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56e1379640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56e1379640 4 set_mon_vals no callback set 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.745+0000 7f56d9ffb640 10 monclient: dump: 2024-03-25T18:26:50.747 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:50.748 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 10 monclient: shutdown 2024-03-25T18:26:50.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 1 -- 172.21.15.176:0/710763929 >> v1:172.21.15.176:6790/0 conn(0x7f56dc14be90 legacy=0x7f56dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 1 -- 172.21.15.176:0/710763929 shutdown_connections 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 1 -- 172.21.15.176:0/710763929 >> 172.21.15.176:0/710763929 conn(0x7f56dc0617a0 msgr2=0x7f56dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 1 -- 172.21.15.176:0/710763929 shutdown_connections 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.746+0000 7f56e1379640 1 -- 172.21.15.176:0/710763929 wait complete. 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 10 monclient: build_initial_monmap 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 10 monclient: monmap: 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:50.751 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 librados: starting msgr at 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 librados: starting objecter 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 Processor -- start 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 -- start start 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 librados: setting wanted keys 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 1 librados: calling monclient init 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.747+0000 7f56e1379640 10 monclient: init 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-a con 0x7f56dc0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-b con 0x7f56dc0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:50.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient(hunting): picked mon.noname-c con 0x7f56dc14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:50.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc1419c0 con 0x7f56dc0b1730 2024-03-25T18:26:50.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc13bbc0 con 0x7f56dc0b4d60 2024-03-25T18:26:50.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc13bd50 con 0x7f56dc14be90 2024-03-25T18:26:50.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:50.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.748+0000 7f56e1379640 10 monclient(hunting): authenticate will time out at 8289.061550s 2024-03-25T18:26:50.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.757+0000 7f56da7fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f56dc0b4d60 0x7f56dc1d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:60486/0 (socket says 172.21.15.176:60486) 2024-03-25T18:26:50.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.757+0000 7f56da7fc640 1 -- 172.21.15.176:0/2525824161 learned_addr learned my addr 172.21.15.176:0/2525824161 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:50.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.758+0000 7f56c37fe640 10 client.?.objecter ms_handle_connect 0x7f56dc0b4d60 2024-03-25T18:26:50.760 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.758+0000 7f56c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.764+0000 7f56c37fe640 10 client.?.objecter ms_handle_connect 0x7f56dc0b1730 2024-03-25T18:26:50.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.764+0000 7f56c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214883282 0 0) 0x7f56dc1419c0 con 0x7f56dc0b1730 2024-03-25T18:26:50.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.770+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c8004eb0 con 0x7f56dc0b1730 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431780450 0 0) 0x7f56dc13bbc0 con 0x7f56dc0b4d60 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc1419c0 con 0x7f56dc0b4d60 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 client.?.objecter ms_handle_connect 0x7f56dc14be90 2024-03-25T18:26:50.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.771+0000 7f56c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:50.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214656947 0 0) 0x7f56dc13bd50 con 0x7f56dc14be90 2024-03-25T18:26:50.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:50.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:50.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:50.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:26:50.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.784+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc13bbc0 con 0x7f56dc14be90 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3411110354 0 0) 0x7f56dc1419c0 con 0x7f56dc0b4d60 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56dc13bd50 con 0x7f56dc0b4d60 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f56d0001950 con 0x7f56dc0b4d60 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): dump: 2024-03-25T18:26:50.789 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889852488 0 0) 0x7f56c8004eb0 con 0x7f56dc0b1730 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56dc1419c0 con 0x7f56dc0b1730 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f56cc0018f0 con 0x7f56dc0b1730 2024-03-25T18:26:50.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.787+0000 7f56c37fe640 10 monclient(hunting): dump: 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:50.791 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733839888 0 0) 0x7f56dc13bd50 con 0x7f56dc0b4d60 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 >> v1:172.21.15.176:6790/0 conn(0x7f56dc14be90 legacy=0x7f56dc1e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 >> v1:172.21.15.176:6789/0 conn(0x7f56dc0b1730 legacy=0x7f56dc1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 1 monclient: found mon.b 2024-03-25T18:26:50.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc13b060 con 0x7f56dc0b4d60 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 10 monclient: _renew_subs 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc1421a0 con 0x7f56dc0b4d60 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 10 monclient: _renew_subs 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:26:50.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc1d8590 con 0x7f56dc0b4d60 2024-03-25T18:26:50.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.801+0000 7f56e1379640 1 librados: init done 2024-03-25T18:26:50.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.802+0000 7f56e1379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:50.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.802+0000 7f56e1379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:50.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.802+0000 7f56e1379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:26:50.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:50.802+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc13b250 con 0x7f56dc0b4d60 2024-03-25T18:26:51.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56d0001d60 con 0x7f56dc0b4d60 2024-03-25T18:26:51.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f56d00049c0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56c37fe640 10 monclient: dump: 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:51.085 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:51.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.082+0000 7f56d97fa640 4 set_mon_vals no callback set 2024-03-25T18:26:51.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.083+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f56d0004be0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.084+0000 7f56daffd640 10 monclient: get_auth_request con 0x7f56c806df70 auth_method 0 2024-03-25T18:26:51.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.084+0000 7f56c37fe640 10 client.4330.objecter ms_handle_connect 0x7f56c806df70 2024-03-25T18:26:51.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.088+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f56d00547b0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.088+0000 7f56c37fe640 10 client.4330.objecter ms_dispatch 0x7f56dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:51.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.088+0000 7f56c37fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:51.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.088+0000 7f56c37fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:51.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.088+0000 7f56c37fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:26:51.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.089+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56d00502c0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.089+0000 7f56c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:51.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.089+0000 7f56c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:51.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.317+0000 7f56e1379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.317+0000 7f56e1379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.317+0000 7f56e1379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:26:51.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.317+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56dc0627b0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.344+0000 7f56c37fe640 1 -- 172.21.15.176:0/2525824161 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f56d00521f0 con 0x7f56dc0b4d60 2024-03-25T18:26:51.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.344+0000 7f56c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:51.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.344+0000 7f56c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:51.346 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:51.346 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 10 librados: watch_flush enter 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 10 librados: watch_flush exit 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 >> v1:172.21.15.176:6800/2598213983 conn(0x7f56c806df70 legacy=0x7f56c8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 10 monclient: shutdown 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 >> v1:172.21.15.186:6789/0 conn(0x7f56dc0b4d60 legacy=0x7f56dc1d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:51.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 shutdown_connections 2024-03-25T18:26:51.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 >> 172.21.15.176:0/2525824161 conn(0x7f56dc0617a0 msgr2=0x7f56dc13c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:51.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.346+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 shutdown_connections 2024-03-25T18:26:51.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.347+0000 7f56e1379640 1 -- 172.21.15.176:0/2525824161 wait complete. 2024-03-25T18:26:51.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:51.347+0000 7f56e1379640 1 librados: shutdown 2024-03-25T18:26:51.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:58.170 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:58.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.282+0000 7f451860d640 10 monclient: get_monmap_and_config 2024-03-25T18:26:58.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.282+0000 7f451860d640 10 monclient: build_initial_monmap 2024-03-25T18:26:58.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.282+0000 7f451860d640 10 monclient: monmap: 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:26:58.283688+0000 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:26:58.283688+0000 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:26:58.284 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.283+0000 7f451860d640 1 Processor -- start 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.283+0000 7f451860d640 1 -- start start 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.283+0000 7f451860d640 10 monclient: init 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient(hunting): picked mon.noname-c con 0x7f451013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient(hunting): picked mon.noname-a con 0x7f45101430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510142520 con 0x7f45101430c0 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b28a0 con 0x7f451013f710 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b2a30 con 0x7f45100a98f0 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.284+0000 7f451860d640 10 monclient(hunting): authenticate will time out at 8296.596458s 2024-03-25T18:26:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.285+0000 7f45163a9640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f45100a98f0 0x7f45100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:49838/0 (socket says 172.21.15.176:49838) 2024-03-25T18:26:58.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.285+0000 7f45163a9640 1 -- 172.21.15.176:0/1045960175 learned_addr learned my addr 172.21.15.176:0/1045960175 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042644884 0 0) 0x7f45100b2a30 con 0x7f45100a98f0 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:26:58.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.306+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc003520 con 0x7f45100a98f0 2024-03-25T18:26:58.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770289429 0 0) 0x7f45100b28a0 con 0x7f451013f710 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45100b2a30 con 0x7f451013f710 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476357881 0 0) 0x7f4510142520 con 0x7f45101430c0 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:26:58.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.308+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45100b28a0 con 0x7f45101430c0 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126485681 0 0) 0x7f45100b2a30 con 0x7f451013f710 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 >> v1:172.21.15.176:6790/0 conn(0x7f45100a98f0 legacy=0x7f45100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 >> v1:172.21.15.176:6789/0 conn(0x7f45101430c0 legacy=0x7f451014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: _finish_hunting 0 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 monclient: found mon.noname-c 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:58.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100b44c0 con 0x7f451013f710 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: _finish_auth 0 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: _check_auth_tickets 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4500001670 con 0x7f451013f710 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f45153a7640 10 monclient: dump: 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:58.322 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:58.323 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:58.323 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:58.323 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.323 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f451860d640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:26:58.323 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.320+0000 7f451860d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45000017d0 con 0x7f451013f710 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 1 -- 172.21.15.176:0/1045960175 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4500001be0 con 0x7f451013f710 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f451860d640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f451860d640 4 set_mon_vals no callback set 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.347+0000 7f45153a7640 10 monclient: dump: 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.348 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:58.349 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 10 monclient: shutdown 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 1 -- 172.21.15.176:0/1045960175 >> v1:172.21.15.186:6789/0 conn(0x7f451013f710 legacy=0x7f45101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 1 -- 172.21.15.176:0/1045960175 shutdown_connections 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 1 -- 172.21.15.176:0/1045960175 >> 172.21.15.176:0/1045960175 conn(0x7f45100b7ea0 msgr2=0x7f45100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:58.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 1 -- 172.21.15.176:0/1045960175 shutdown_connections 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.348+0000 7f451860d640 1 -- 172.21.15.176:0/1045960175 wait complete. 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 10 monclient: build_initial_monmap 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 10 monclient: monmap: 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 librados: starting msgr at 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 librados: starting objecter 2024-03-25T18:26:58.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 Processor -- start 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 -- start start 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 librados: setting wanted keys 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 1 librados: calling monclient init 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.349+0000 7f451860d640 10 monclient: init 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient(hunting): picked mon.noname-c con 0x7f45100a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient(hunting): picked mon.noname-a con 0x7f451013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient(hunting): picked mon.noname-b con 0x7f45101430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451013b180 con 0x7f451013f710 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45101db890 con 0x7f45101430c0 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b7b90 con 0x7f45100a98f0 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:58.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.350+0000 7f451860d640 10 monclient(hunting): authenticate will time out at 8296.662321s 2024-03-25T18:26:58.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.354+0000 7f4515ba8640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f451013f710 0x7f45101daa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:47700/0 (socket says 172.21.15.176:47700) 2024-03-25T18:26:58.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.354+0000 7f4515ba8640 1 -- 172.21.15.176:0/978337346 learned_addr learned my addr 172.21.15.176:0/978337346 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:26:58.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.356+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f451013f710 2024-03-25T18:26:58.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.356+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.363+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f45100a98f0 2024-03-25T18:26:58.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.363+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831854224 0 0) 0x7f45100b7b90 con 0x7f45100a98f0 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:26:58.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.371+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f4004eb0 con 0x7f45100a98f0 2024-03-25T18:26:58.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.375+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f45101430c0 2024-03-25T18:26:58.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.375+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1837942902 0 0) 0x7f44f4004eb0 con 0x7f45100a98f0 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45100b7b90 con 0x7f45100a98f0 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f450c001cd0 con 0x7f45100a98f0 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:58.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.386+0000 7f4506ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6450610 0 0) 0x7f45101db890 con 0x7f45101430c0 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:26:58.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.391+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f4004eb0 con 0x7f45101430c0 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3500458482 0 0) 0x7f45100b7b90 con 0x7f45100a98f0 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 >> v1:172.21.15.186:6789/0 conn(0x7f45101430c0 legacy=0x7f45101db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 >> v1:172.21.15.176:6789/0 conn(0x7f451013f710 legacy=0x7f45101daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 1 monclient: found mon.c 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:58.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451013b3f0 con 0x7f45100a98f0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f4506ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 10 monclient: _renew_subs 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451013a1b0 con 0x7f45100a98f0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 10 monclient: _renew_subs 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f451013b9f0 con 0x7f45100a98f0 2024-03-25T18:26:58.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.394+0000 7f451860d640 1 librados: init done 2024-03-25T18:26:58.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.395+0000 7f451860d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.395+0000 7f451860d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.395+0000 7f451860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:58.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.395+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45100b3ef0 con 0x7f45100a98f0 2024-03-25T18:26:58.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f450c0020e0 con 0x7f45100a98f0 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f450c004d40 con 0x7f45100a98f0 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4506ffd640 10 monclient: dump: 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:26:58.700 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:26:58.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.699+0000 7f4514ba6640 4 set_mon_vals no callback set 2024-03-25T18:26:58.702 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.700+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f450c004f60 con 0x7f45100a98f0 2024-03-25T18:26:58.702 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.700+0000 7f4515ba8640 10 monclient: get_auth_request con 0x7f44f406dd90 auth_method 0 2024-03-25T18:26:58.702 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.700+0000 7f4506ffd640 10 client.4349.objecter ms_handle_connect 0x7f44f406dd90 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f450c054ad0 con 0x7f45100a98f0 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 10 client.4349.objecter ms_dispatch 0x7f4510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f450c050410 con 0x7f45100a98f0 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:58.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.707+0000 7f4506ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:58.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.936+0000 7f451860d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:58.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.936+0000 7f451860d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:58.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.936+0000 7f451860d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:26:58.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:58.936+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45100627b0 con 0x7f45100a98f0 2024-03-25T18:26:59.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.044+0000 7f4506ffd640 1 -- 172.21.15.176:0/978337346 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f450c052510 con 0x7f45100a98f0 2024-03-25T18:26:59.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.044+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:59.045 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.044+0000 7f4506ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:59.045 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:26:59.045 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.046+0000 7f451860d640 10 librados: watch_flush enter 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.046+0000 7f451860d640 10 librados: watch_flush exit 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.046+0000 7f451860d640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.046+0000 7f451860d640 10 client.4349.objecter successfully canceled tick 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 >> v1:172.21.15.176:6800/2598213983 conn(0x7f44f406dd90 legacy=0x7f44f4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 10 monclient: shutdown 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:59.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 >> v1:172.21.15.176:6790/0 conn(0x7f45100a98f0 legacy=0x7f45100b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 shutdown_connections 2024-03-25T18:26:59.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 >> 172.21.15.176:0/978337346 conn(0x7f45100b7ea0 msgr2=0x7f4510074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:59.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 shutdown_connections 2024-03-25T18:26:59.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 -- 172.21.15.176:0/978337346 wait complete. 2024-03-25T18:26:59.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:26:59.047+0000 7f451860d640 1 librados: shutdown 2024-03-25T18:26:59.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:05.867 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.987+0000 7f7206c15640 10 monclient: get_monmap_and_config 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.987+0000 7f7206c15640 10 monclient: build_initial_monmap 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.987+0000 7f7206c15640 10 monclient: monmap: 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:05.988192+0000 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:05.988192+0000 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:05.988 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:05.989 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:05.989 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:05.989 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:05.989 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 1 Processor -- start 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 1 -- start start 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient: init 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-b con 0x7f72000b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:05.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-a con 0x7f72000b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-c con 0x7f720014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72000af940 con 0x7f72000b1730 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72000b0a70 con 0x7f720014be90 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72000b0d50 con 0x7f72000b4d60 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:05.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.988+0000 7f7206c15640 10 monclient(hunting): authenticate will time out at 8304.300720s 2024-03-25T18:27:05.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.998+0000 7f72051b2640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f720014be90 0x7f720014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:47400/0 (socket says 172.21.15.176:47400) 2024-03-25T18:27:05.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:05.998+0000 7f72051b2640 1 -- 172.21.15.176:0/336873288 learned_addr learned my addr 172.21.15.176:0/336873288 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127598439 0 0) 0x7f72000b0d50 con 0x7f72000b4d60 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:27:06.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.022+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f0003520 con 0x7f72000b4d60 2024-03-25T18:27:06.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169027651 0 0) 0x7f72000af940 con 0x7f72000b1730 2024-03-25T18:27:06.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72000b0d50 con 0x7f72000b1730 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889008344 0 0) 0x7f72000b0a70 con 0x7f720014be90 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:27:06.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.024+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72000af940 con 0x7f720014be90 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.032+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404989699 0 0) 0x7f72000b0d50 con 0x7f72000b1730 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 >> v1:172.21.15.176:6790/0 conn(0x7f72000b4d60 legacy=0x7f720013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 >> v1:172.21.15.186:6789/0 conn(0x7f720014be90 legacy=0x7f720014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000b1030 con 0x7f72000b1730 2024-03-25T18:27:06.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f71ec001670 con 0x7f72000b1730 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f71ff7fe640 10 monclient: dump: 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:06.035 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f7206c15640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:27:06.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.033+0000 7f7206c15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ec0017d0 con 0x7f72000b1730 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 1 -- 172.21.15.176:0/336873288 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f71ec001be0 con 0x7f72000b1730 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f7206c15640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f7206c15640 4 set_mon_vals no callback set 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.051+0000 7f71ff7fe640 10 monclient: dump: 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.052 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:06.053 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:06.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 10 monclient: shutdown 2024-03-25T18:27:06.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:06.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 1 -- 172.21.15.176:0/336873288 >> v1:172.21.15.176:6789/0 conn(0x7f72000b1730 legacy=0x7f72000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 1 -- 172.21.15.176:0/336873288 shutdown_connections 2024-03-25T18:27:06.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 1 -- 172.21.15.176:0/336873288 >> 172.21.15.176:0/336873288 conn(0x7f72000617a0 msgr2=0x7f7200061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 1 -- 172.21.15.176:0/336873288 shutdown_connections 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.052+0000 7f7206c15640 1 -- 172.21.15.176:0/336873288 wait complete. 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 10 monclient: build_initial_monmap 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 10 monclient: monmap: 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:06.056 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 librados: starting msgr at 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 librados: starting objecter 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 Processor -- start 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 -- start start 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 librados: setting wanted keys 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 1 librados: calling monclient init 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.053+0000 7f7206c15640 10 monclient: init 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-a con 0x7f72000b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-b con 0x7f72000b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient(hunting): picked mon.noname-c con 0x7f720014be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720013a270 con 0x7f72000b1730 2024-03-25T18:27:06.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720013b490 con 0x7f72000b4d60 2024-03-25T18:27:06.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7200139110 con 0x7f720014be90 2024-03-25T18:27:06.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:06.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.054+0000 7f7206c15640 10 monclient(hunting): authenticate will time out at 8304.366444s 2024-03-25T18:27:06.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.060+0000 7f71fffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f72000b4d60 0x7f72001db090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:47440/0 (socket says 172.21.15.176:47440) 2024-03-25T18:27:06.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.060+0000 7f71fffff640 1 -- 172.21.15.176:0/1588420023 learned_addr learned my addr 172.21.15.176:0/1588420023 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:06.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.061+0000 7f71fd7fa640 10 client.?.objecter ms_handle_connect 0x7f72000b4d60 2024-03-25T18:27:06.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.061+0000 7f71fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:06.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110544009 0 0) 0x7f720013b490 con 0x7f72000b4d60 2024-03-25T18:27:06.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:27:06.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.072+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e4004eb0 con 0x7f72000b4d60 2024-03-25T18:27:06.077 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.076+0000 7f71fd7fa640 10 client.?.objecter ms_handle_connect 0x7f720014be90 2024-03-25T18:27:06.077 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.076+0000 7f71fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:06.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.079+0000 7f71fd7fa640 10 client.?.objecter ms_handle_connect 0x7f72000b1730 2024-03-25T18:27:06.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.079+0000 7f71fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593860945 0 0) 0x7f71e4004eb0 con 0x7f72000b4d60 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f720013b490 con 0x7f72000b4d60 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f71ec001670 con 0x7f72000b4d60 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.087 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:06.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.086+0000 7f71fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:06.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770405581 0 0) 0x7f720013a270 con 0x7f72000b1730 2024-03-25T18:27:06.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:06.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:06.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:06.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:27:06.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.090+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e4004eb0 con 0x7f72000b1730 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1547084043 0 0) 0x7f720013b490 con 0x7f72000b4d60 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 >> v1:172.21.15.176:6790/0 conn(0x7f720014be90 legacy=0x7f7200132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 >> v1:172.21.15.176:6789/0 conn(0x7f72000b1730 legacy=0x7f7200139b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 1 monclient: found mon.b 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72001dab70 con 0x7f72000b4d60 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f71fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:06.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 10 monclient: _renew_subs 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7200143900 con 0x7f72000b4d60 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 10 monclient: _renew_subs 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72001db7a0 con 0x7f72000b4d60 2024-03-25T18:27:06.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.091+0000 7f7206c15640 1 librados: init done 2024-03-25T18:27:06.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.092+0000 7f7206c15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.092+0000 7f7206c15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.092+0000 7f7206c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:06.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.092+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72001dba40 con 0x7f72000b4d60 2024-03-25T18:27:06.365 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.364+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ec0038a0 con 0x7f72000b4d60 2024-03-25T18:27:06.365 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.364+0000 7f71fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:06.365 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.364+0000 7f71feffd640 4 set_mon_vals no callback set 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.375+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f71ec003a90 con 0x7f72000b4d60 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.375+0000 7f71fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.375+0000 7f71fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.375+0000 7f71fd7fa640 10 monclient: dump: 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:06.376 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:06.377 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:06.378 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.376+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f71ec003c70 con 0x7f72000b4d60 2024-03-25T18:27:06.378 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.377+0000 7f72049b1640 10 monclient: get_auth_request con 0x7f71e406dd90 auth_method 0 2024-03-25T18:27:06.378 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.377+0000 7f71fd7fa640 10 client.4342.objecter ms_handle_connect 0x7f71e406dd90 2024-03-25T18:27:06.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.382+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f71ec0547d0 con 0x7f72000b4d60 2024-03-25T18:27:06.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.382+0000 7f71fd7fa640 10 client.4342.objecter ms_dispatch 0x7f7200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:06.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.382+0000 7f71fd7fa640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:06.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.382+0000 7f71fd7fa640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:06.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.382+0000 7f71fd7fa640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:27:06.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.383+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71ec0502e0 con 0x7f72000b4d60 2024-03-25T18:27:06.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.383+0000 7f71fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:06.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.383+0000 7f71fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:06.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.597+0000 7f7206c15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.597+0000 7f7206c15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.597+0000 7f7206c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:06.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.597+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72001dad00 con 0x7f72000b4d60 2024-03-25T18:27:06.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.630+0000 7f71fd7fa640 1 -- 172.21.15.176:0/1588420023 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f71ec052210 con 0x7f72000b4d60 2024-03-25T18:27:06.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.630+0000 7f71fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:06.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.630+0000 7f71fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:06.632 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:06.632 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.632+0000 7f7206c15640 10 librados: watch_flush enter 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.632+0000 7f7206c15640 10 librados: watch_flush exit 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.632+0000 7f7206c15640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.632+0000 7f7206c15640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.632+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 >> v1:172.21.15.176:6800/2598213983 conn(0x7f71e406dd90 legacy=0x7f71e4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 10 monclient: shutdown 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:06.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 >> v1:172.21.15.186:6789/0 conn(0x7f72000b4d60 legacy=0x7f72001db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:06.636 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 shutdown_connections 2024-03-25T18:27:06.636 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 >> 172.21.15.176:0/1588420023 conn(0x7f72000617a0 msgr2=0x7f72000b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:06.636 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 shutdown_connections 2024-03-25T18:27:06.636 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 -- 172.21.15.176:0/1588420023 wait complete. 2024-03-25T18:27:06.636 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:06.633+0000 7f7206c15640 1 librados: shutdown 2024-03-25T18:27:06.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:07.771 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:09.527 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.828 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:09.584 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.856 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:09.612 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.894 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:09.628 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.899 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:09.655 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.901 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:09.636 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.910 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:09.666 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.926 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:09.661 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.947 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:09.704 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.453 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:13.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.567+0000 7f7ea2160640 10 monclient: get_monmap_and_config 2024-03-25T18:27:13.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.567+0000 7f7ea2160640 10 monclient: build_initial_monmap 2024-03-25T18:27:13.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.567+0000 7f7ea2160640 10 monclient: monmap: 2024-03-25T18:27:13.568 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:13.568551+0000 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:13.568551+0000 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:13.569 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.568+0000 7f7ea2160640 1 Processor -- start 2024-03-25T18:27:13.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.568+0000 7f7ea2160640 1 -- start start 2024-03-25T18:27:13.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.568+0000 7f7ea2160640 10 monclient: init 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-b con 0x7f7e9c0b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-a con 0x7f7e9c0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-c con 0x7f7e9c14be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c0af940 con 0x7f7e9c0b1730 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c0b0a70 con 0x7f7e9c14be90 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c0b0d50 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:13.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.569+0000 7f7ea2160640 10 monclient(hunting): authenticate will time out at 8311.881357s 2024-03-25T18:27:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.579+0000 7f7e9affd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7e9c0b1730 0x7f7e9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:53530/0 (socket says 172.21.15.176:53530) 2024-03-25T18:27:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.579+0000 7f7e9affd640 1 -- 172.21.15.176:0/2298172453 learned_addr learned my addr 172.21.15.176:0/2298172453 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978112661 0 0) 0x7f7e9c0af940 con 0x7f7e9c0b1730 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:27:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.589+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e84003520 con 0x7f7e9c0b1730 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142175298 0 0) 0x7f7e9c0b0a70 con 0x7f7e9c14be90 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:27:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.591+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c0af940 con 0x7f7e9c14be90 2024-03-25T18:27:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915278515 0 0) 0x7f7e9c0b0d50 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:27:13.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.592+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c0b0a70 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 617249747 0 0) 0x7f7e84003520 con 0x7f7e9c0b1730 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 >> v1:172.21.15.176:6790/0 conn(0x7f7e9c0b4d60 legacy=0x7f7e9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 >> v1:172.21.15.186:6789/0 conn(0x7f7e9c14be90 legacy=0x7f7e9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:13.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c0b1030 con 0x7f7e9c0b1730 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7e90001690 con 0x7f7e9c0b1730 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7e9a7fc640 10 monclient: dump: 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.601 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7ea2160640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:27:13.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.599+0000 7f7ea2160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:13.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e900017f0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 1 -- 172.21.15.176:0/2298172453 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7e90001c00 con 0x7f7e9c0b1730 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7ea2160640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7ea2160640 4 set_mon_vals no callback set 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.615+0000 7f7e9a7fc640 10 monclient: dump: 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:13.617 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:13.618 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 10 monclient: shutdown 2024-03-25T18:27:13.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 1 -- 172.21.15.176:0/2298172453 >> v1:172.21.15.176:6789/0 conn(0x7f7e9c0b1730 legacy=0x7f7e9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 1 -- 172.21.15.176:0/2298172453 shutdown_connections 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 1 -- 172.21.15.176:0/2298172453 >> 172.21.15.176:0/2298172453 conn(0x7f7e9c0617a0 msgr2=0x7f7e9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 1 -- 172.21.15.176:0/2298172453 shutdown_connections 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.616+0000 7f7ea2160640 1 -- 172.21.15.176:0/2298172453 wait complete. 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.617+0000 7f7ea2160640 10 monclient: build_initial_monmap 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.617+0000 7f7ea2160640 10 monclient: monmap: 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:13.621 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 librados: starting msgr at 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 librados: starting objecter 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 Processor -- start 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 -- start start 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 librados: setting wanted keys 2024-03-25T18:27:13.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 1 librados: calling monclient init 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.618+0000 7f7ea2160640 10 monclient: init 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-c con 0x7f7e9c0b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-b con 0x7f7e9c0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient(hunting): picked mon.noname-a con 0x7f7e9c14be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c134f90 con 0x7f7e9c14be90 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c140f00 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c1411e0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:13.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.619+0000 7f7ea2160640 10 monclient(hunting): authenticate will time out at 8311.931101s 2024-03-25T18:27:13.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.627+0000 7f7e9bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7e9c14be90 0x7f7e9c134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:53582/0 (socket says 172.21.15.176:53582) 2024-03-25T18:27:13.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.628+0000 7f7e9bfff640 1 -- 172.21.15.176:0/1163933763 learned_addr learned my addr 172.21.15.176:0/1163933763 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:13.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.630+0000 7f7e83fff640 10 client.?.objecter ms_handle_connect 0x7f7e9c14be90 2024-03-25T18:27:13.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.630+0000 7f7e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.637+0000 7f7e83fff640 10 client.?.objecter ms_handle_connect 0x7f7e9c0b4d60 2024-03-25T18:27:13.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.637+0000 7f7e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.638+0000 7f7e83fff640 10 client.?.objecter ms_handle_connect 0x7f7e9c0b1730 2024-03-25T18:27:13.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.638+0000 7f7e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:13.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286852321 0 0) 0x7f7e9c1411e0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:27:13.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.647+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e88004eb0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252598569 0 0) 0x7f7e9c134f90 con 0x7f7e9c14be90 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:27:13.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.650+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c1411e0 con 0x7f7e9c14be90 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3334367220 0 0) 0x7f7e88004eb0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e9c134f90 con 0x7f7e9c0b1730 2024-03-25T18:27:13.659 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7e8c0018f0 con 0x7f7e9c0b1730 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 10 monclient(hunting): dump: 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.658+0000 7f7e83fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:13.663 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503207037 0 0) 0x7f7e9c140f00 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:13.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:13.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:13.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:27:13.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.663+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e88004eb0 con 0x7f7e9c0b4d60 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3910693631 0 0) 0x7f7e9c1411e0 con 0x7f7e9c14be90 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e9c140f00 con 0x7f7e9c14be90 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7e94001c10 con 0x7f7e9c14be90 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.673+0000 7f7e83fff640 10 monclient(hunting): dump: 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:13.674 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:13.675 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:13.681 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917200722 0 0) 0x7f7e9c134f90 con 0x7f7e9c0b1730 2024-03-25T18:27:13.681 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 >> v1:172.21.15.186:6789/0 conn(0x7f7e9c0b4d60 legacy=0x7f7e9c134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.681 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 >> v1:172.21.15.176:6789/0 conn(0x7f7e9c14be90 legacy=0x7f7e9c134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:13.681 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:13.681 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 1 monclient: found mon.c 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c138190 con 0x7f7e9c0b1730 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 10 monclient: _renew_subs 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c1df950 con 0x7f7e9c0b1730 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 10 monclient: _renew_subs 2024-03-25T18:27:13.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:13.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c138870 con 0x7f7e9c0b1730 2024-03-25T18:27:13.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.680+0000 7f7ea2160640 1 librados: init done 2024-03-25T18:27:13.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.681+0000 7f7ea2160640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:13.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.681+0000 7f7ea2160640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:13.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.681+0000 7f7ea2160640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:13.684 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:13.681+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c0627b0 con 0x7f7e9c0b1730 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.003+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e8c003610 con 0x7f7e9c0b1730 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7e8c003920 con 0x7f7e9c0b1730 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 10 monclient: dump: 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:14.005 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e99ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:14.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.004+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f7e8c019930 con 0x7f7e9c0b1730 2024-03-25T18:27:14.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.005+0000 7f7e9affd640 10 monclient: get_auth_request con 0x7f7e8806dfb0 auth_method 0 2024-03-25T18:27:14.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.005+0000 7f7e83fff640 10 client.4361.objecter ms_handle_connect 0x7f7e8806dfb0 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7e8c0535f0 con 0x7f7e9c0b1730 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 10 client.4361.objecter ms_dispatch 0x7f7e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e8c04ef30 con 0x7f7e9c0b1730 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:14.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.011+0000 7f7e83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:14.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.244+0000 7f7ea2160640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.244+0000 7f7ea2160640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.244+0000 7f7ea2160640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:14.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.244+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e9c138320 con 0x7f7e9c0b1730 2024-03-25T18:27:14.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.314+0000 7f7e83fff640 1 -- 172.21.15.176:0/1163933763 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7e8c051030 con 0x7f7e9c0b1730 2024-03-25T18:27:14.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.314+0000 7f7e83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:14.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.314+0000 7f7e83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:14.316 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:14.316 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:14.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 10 librados: watch_flush enter 2024-03-25T18:27:14.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 10 librados: watch_flush exit 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 10 client.4361.objecter successfully canceled tick 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 >> v1:172.21.15.176:6800/2598213983 conn(0x7f7e8806dfb0 legacy=0x7f7e88070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 10 monclient: shutdown 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 >> v1:172.21.15.176:6790/0 conn(0x7f7e9c0b1730 legacy=0x7f7e9c1df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 shutdown_connections 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 >> 172.21.15.176:0/1163933763 conn(0x7f7e9c0617a0 msgr2=0x7f7e9c13c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 shutdown_connections 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 -- 172.21.15.176:0/1163933763 wait complete. 2024-03-25T18:27:14.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:14.317+0000 7f7ea2160640 1 librados: shutdown 2024-03-25T18:27:14.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:21.138 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:21.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.250+0000 7f4399a96640 10 monclient: get_monmap_and_config 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.250+0000 7f4399a96640 10 monclient: build_initial_monmap 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.250+0000 7f4399a96640 10 monclient: monmap: 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:21.251772+0000 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:21.251772+0000 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:21.252 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:21.253 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.251+0000 7f4399a96640 1 Processor -- start 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.251+0000 7f4399a96640 1 -- start start 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.251+0000 7f4399a96640 10 monclient: init 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-c con 0x7f43940a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-a con 0x7f439413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-b con 0x7f43941430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394142520 con 0x7f439413f710 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940b28a0 con 0x7f43940a98f0 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940b2a30 con 0x7f43941430c0 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:21.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.252+0000 7f4399a96640 10 monclient(hunting): authenticate will time out at 8319.564422s 2024-03-25T18:27:21.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.256+0000 7f43937fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f43940a98f0 0x7f43940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:59748/0 (socket says 172.21.15.176:59748) 2024-03-25T18:27:21.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.256+0000 7f43937fe640 1 -- 172.21.15.176:0/3939676140 learned_addr learned my addr 172.21.15.176:0/3939676140 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078592916 0 0) 0x7f43940b28a0 con 0x7f43940a98f0 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:27:21.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.263+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c003520 con 0x7f43940a98f0 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320735828 0 0) 0x7f4394142520 con 0x7f439413f710 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:27:21.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.269+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43940b28a0 con 0x7f439413f710 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799195617 0 0) 0x7f437c003520 con 0x7f43940a98f0 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 >> v1:172.21.15.176:6790/0 conn(0x7f43941430c0 legacy=0x7f439414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 >> v1:172.21.15.176:6789/0 conn(0x7f439413f710 legacy=0x7f43941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43940b44c0 con 0x7f43940a98f0 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:21.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4384001690 con 0x7f43940a98f0 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f4399a96640 5 monclient: authenticate success, global_id 4351 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f4399a96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f43927fc640 10 monclient: dump: 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.276 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:21.277 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:21.277 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:21.277 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.274+0000 7f4399a96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:21.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4384001870 con 0x7f43940a98f0 2024-03-25T18:27:21.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:21.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 1 -- 172.21.15.176:0/3939676140 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4384001c80 con 0x7f43940a98f0 2024-03-25T18:27:21.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f4399a96640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f4399a96640 4 set_mon_vals no callback set 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.282+0000 7f43927fc640 10 monclient: dump: 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:21.284 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 10 monclient: shutdown 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 1 -- 172.21.15.176:0/3939676140 >> v1:172.21.15.186:6789/0 conn(0x7f43940a98f0 legacy=0x7f43940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 1 -- 172.21.15.176:0/3939676140 shutdown_connections 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 1 -- 172.21.15.176:0/3939676140 >> 172.21.15.176:0/3939676140 conn(0x7f43940b7ea0 msgr2=0x7f43940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 1 -- 172.21.15.176:0/3939676140 shutdown_connections 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.283+0000 7f4399a96640 1 -- 172.21.15.176:0/3939676140 wait complete. 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.284+0000 7f4399a96640 10 monclient: build_initial_monmap 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.284+0000 7f4399a96640 10 monclient: monmap: 2024-03-25T18:27:21.286 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 librados: starting msgr at 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 librados: starting objecter 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 Processor -- start 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 -- start start 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 librados: setting wanted keys 2024-03-25T18:27:21.287 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 1 librados: calling monclient init 2024-03-25T18:27:21.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.285+0000 7f4399a96640 10 monclient: init 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-c con 0x7f43940a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-b con 0x7f439413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient(hunting): picked mon.noname-a con 0x7f43941430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940aca40 con 0x7f43941430c0 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940abf10 con 0x7f439413f710 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394138f60 con 0x7f43940a98f0 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:21.289 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.286+0000 7f4399a96640 10 monclient(hunting): authenticate will time out at 8319.598379s 2024-03-25T18:27:21.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.293+0000 7f43937fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f43940a98f0 0x7f4394138690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:47978/0 (socket says 172.21.15.176:47978) 2024-03-25T18:27:21.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.293+0000 7f43937fe640 1 -- 172.21.15.176:0/3811149974 learned_addr learned my addr 172.21.15.176:0/3811149974 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:21.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.293+0000 7f437bfff640 10 client.?.objecter ms_handle_connect 0x7f43940a98f0 2024-03-25T18:27:21.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.293+0000 7f437bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.304 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.304+0000 7f437bfff640 10 client.?.objecter ms_handle_connect 0x7f439413f710 2024-03-25T18:27:21.304 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.304+0000 7f437bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.305+0000 7f437bfff640 10 client.?.objecter ms_handle_connect 0x7f43941430c0 2024-03-25T18:27:21.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.305+0000 7f437bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:21.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33164618 0 0) 0x7f4394138f60 con 0x7f43940a98f0 2024-03-25T18:27:21.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:27:21.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.309+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4380004eb0 con 0x7f43940a98f0 2024-03-25T18:27:21.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226967527 0 0) 0x7f43940abf10 con 0x7f439413f710 2024-03-25T18:27:21.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:27:21.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.312+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4394138f60 con 0x7f439413f710 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410136052 0 0) 0x7f43940aca40 con 0x7f43941430c0 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:27:21.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.314+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43940abf10 con 0x7f43941430c0 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323251003 0 0) 0x7f4394138f60 con 0x7f439413f710 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43940aca40 con 0x7f439413f710 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4388001930 con 0x7f439413f710 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 10 monclient(hunting): dump: 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:21.325 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:21.326 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:21.326 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.326 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.324+0000 7f437bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:21.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842369527 0 0) 0x7f4380004eb0 con 0x7f43940a98f0 2024-03-25T18:27:21.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4394138f60 con 0x7f43940a98f0 2024-03-25T18:27:21.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4384001c40 con 0x7f43940a98f0 2024-03-25T18:27:21.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.326+0000 7f437bfff640 10 monclient(hunting): dump: 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:21.328 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.334 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2632504995 0 0) 0x7f43940aca40 con 0x7f439413f710 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 >> v1:172.21.15.176:6790/0 conn(0x7f43940a98f0 legacy=0x7f4394138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 >> v1:172.21.15.176:6789/0 conn(0x7f43941430c0 legacy=0x7f43941e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 1 monclient: found mon.b 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394139990 con 0x7f439413f710 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f437bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:27:21.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 10 monclient: _renew_subs 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4394137bd0 con 0x7f439413f710 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 10 monclient: _renew_subs 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43941365e0 con 0x7f439413f710 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.333+0000 7f4399a96640 1 librados: init done 2024-03-25T18:27:21.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.334+0000 7f4399a96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.334+0000 7f4399a96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.334+0000 7f4399a96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.334+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43940b17e0 con 0x7f439413f710 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.339+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4388001d60 con 0x7f439413f710 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.339+0000 7f437bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:21.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.339+0000 7f4391ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:21.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.608+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4388003610 con 0x7f439413f710 2024-03-25T18:27:21.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.608+0000 7f437bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:21.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.608+0000 7f437bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.608+0000 7f437bfff640 10 monclient: dump: 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:21.610 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:21.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.609+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f4388003830 con 0x7f439413f710 2024-03-25T18:27:21.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.609+0000 7f43937fe640 10 monclient: get_auth_request con 0x7f438006df10 auth_method 0 2024-03-25T18:27:21.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.610+0000 7f437bfff640 10 client.4354.objecter ms_handle_connect 0x7f438006df10 2024-03-25T18:27:21.614 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.613+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f4388053440 con 0x7f439413f710 2024-03-25T18:27:21.614 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.613+0000 7f437bfff640 10 client.4354.objecter ms_dispatch 0x7f4394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:21.614 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.613+0000 7f437bfff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:21.614 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.613+0000 7f437bfff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:21.614 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.613+0000 7f437bfff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:27:21.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.615+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f438804ef50 con 0x7f439413f710 2024-03-25T18:27:21.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.615+0000 7f437bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:21.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.615+0000 7f437bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:21.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.855+0000 7f4399a96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:21.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.855+0000 7f4399a96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:21.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.855+0000 7f4399a96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:21.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.855+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4394138da0 con 0x7f439413f710 2024-03-25T18:27:21.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.874+0000 7f437bfff640 1 -- 172.21.15.176:0/3811149974 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f4388050e80 con 0x7f439413f710 2024-03-25T18:27:21.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.874+0000 7f437bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:21.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.874+0000 7f437bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:21.875 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:21.875 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.876+0000 7f4399a96640 10 librados: watch_flush enter 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.876+0000 7f4399a96640 10 librados: watch_flush exit 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.876+0000 7f4399a96640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.876+0000 7f4399a96640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 >> v1:172.21.15.176:6800/2598213983 conn(0x7f438006df10 legacy=0x7f43800703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 10 monclient: shutdown 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 >> v1:172.21.15.186:6789/0 conn(0x7f439413f710 legacy=0x7f43940ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 shutdown_connections 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 >> 172.21.15.176:0/3811149974 conn(0x7f43940b7ea0 msgr2=0x7f4394074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 shutdown_connections 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 -- 172.21.15.176:0/3811149974 wait complete. 2024-03-25T18:27:21.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:21.877+0000 7f4399a96640 1 librados: shutdown 2024-03-25T18:27:21.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:27.865 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:29.621 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.918 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:29.674 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.963 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:29.719 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.971 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:29.706 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.977 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:29.711 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.985 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:29.742 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.992 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:29.748 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:28.010 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:29.744 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:28.017 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:29.774 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:28.697 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:28.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.810+0000 7fa4aa967640 10 monclient: get_monmap_and_config 2024-03-25T18:27:28.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.810+0000 7fa4aa967640 10 monclient: build_initial_monmap 2024-03-25T18:27:28.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.810+0000 7fa4aa967640 10 monclient: monmap: 2024-03-25T18:27:28.811 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:28.811575+0000 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:28.811575+0000 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:28.812 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.813 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.811+0000 7fa4aa967640 1 Processor -- start 2024-03-25T18:27:28.813 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.811+0000 7fa4aa967640 1 -- start start 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.811+0000 7fa4aa967640 10 monclient: init 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4142520 con 0x7fa4a40a98f0 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40b28a0 con 0x7fa4a41430c0 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40b2a30 con 0x7fa4a413f710 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:28.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.812+0000 7fa4aa967640 10 monclient(hunting): authenticate will time out at 8327.124493s 2024-03-25T18:27:28.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.821+0000 7fa4a37fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa4a413f710 0x7fa4a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:42798/0 (socket says 172.21.15.176:42798) 2024-03-25T18:27:28.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.821+0000 7fa4a37fe640 1 -- 172.21.15.176:0/3094765474 learned_addr learned my addr 172.21.15.176:0/3094765474 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444661063 0 0) 0x7fa4a40b2a30 con 0x7fa4a413f710 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:27:28.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.836+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c003520 con 0x7fa4a413f710 2024-03-25T18:27:28.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217109621 0 0) 0x7fa4a4142520 con 0x7fa4a40a98f0 2024-03-25T18:27:28.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:27:28.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.839+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a40b2a30 con 0x7fa4a40a98f0 2024-03-25T18:27:28.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.840+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814009625 0 0) 0x7fa4a40b28a0 con 0x7fa4a41430c0 2024-03-25T18:27:28.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.840+0000 7fa4a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.840+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.840+0000 7fa4a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.840+0000 7fa4a2ffd640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:27:28.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.841+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4142520 con 0x7fa4a41430c0 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4188622203 0 0) 0x7fa48c003520 con 0x7fa4a413f710 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 >> v1:172.21.15.186:6789/0 conn(0x7fa4a41430c0 legacy=0x7fa4a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 >> v1:172.21.15.176:6789/0 conn(0x7fa4a40a98f0 legacy=0x7fa4a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40b44c0 con 0x7fa4a413f710 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:28.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa498001690 con 0x7fa4a413f710 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.848+0000 7fa4a2ffd640 10 monclient: dump: 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.851 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:28.852 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:28.852 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:28.852 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.849+0000 7fa4aa967640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:27:28.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.849+0000 7fa4aa967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:28.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.859+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4980017f0 con 0x7fa4a413f710 2024-03-25T18:27:28.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:28.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4a2ffd640 1 -- 172.21.15.176:0/3094765474 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa498001c00 con 0x7fa4a413f710 2024-03-25T18:27:28.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4aa967640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4aa967640 4 set_mon_vals no callback set 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4a2ffd640 10 monclient: dump: 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.861 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:28.862 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:28.862 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:28.862 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4aa967640 10 monclient: shutdown 2024-03-25T18:27:28.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4aa967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:28.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.860+0000 7fa4aa967640 1 -- 172.21.15.176:0/3094765474 >> v1:172.21.15.176:6790/0 conn(0x7fa4a413f710 legacy=0x7fa4a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:28.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 1 -- 172.21.15.176:0/3094765474 shutdown_connections 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 1 -- 172.21.15.176:0/3094765474 >> 172.21.15.176:0/3094765474 conn(0x7fa4a40b7ea0 msgr2=0x7fa4a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 1 -- 172.21.15.176:0/3094765474 shutdown_connections 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 1 -- 172.21.15.176:0/3094765474 wait complete. 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 10 monclient: build_initial_monmap 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.861+0000 7fa4aa967640 10 monclient: monmap: 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:28.865 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 librados: starting msgr at 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 librados: starting objecter 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 Processor -- start 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 -- start start 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 librados: setting wanted keys 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.862+0000 7fa4aa967640 1 librados: calling monclient init 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient: init 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4137340 con 0x7fa4a41430c0 2024-03-25T18:27:28.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4135260 con 0x7fa4a413f710 2024-03-25T18:27:28.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4135fe0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:28.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.863+0000 7fa4aa967640 10 monclient(hunting): authenticate will time out at 8327.175619s 2024-03-25T18:27:28.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.869+0000 7fa4a8f04640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa4a41430c0 0x7fa4a41d78e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:52528/0 (socket says 172.21.15.176:52528) 2024-03-25T18:27:28.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.869+0000 7fa4a8f04640 1 -- 172.21.15.176:0/1490074560 learned_addr learned my addr 172.21.15.176:0/1490074560 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:28.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.872+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a41430c0 2024-03-25T18:27:28.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.872+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:28.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.878+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a40a98f0 2024-03-25T18:27:28.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.878+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:28.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.884+0000 7fa4a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa4a413f710 2024-03-25T18:27:28.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.884+0000 7fa4a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21170593 0 0) 0x7fa4a4135fe0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:27:28.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.890+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490004eb0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722906690 0 0) 0x7fa4a4135260 con 0x7fa4a413f710 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:27:28.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.897+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4135fe0 con 0x7fa4a413f710 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140341062 0 0) 0x7fa4a4137340 con 0x7fa4a41430c0 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:27:28.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.900+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4135260 con 0x7fa4a41430c0 2024-03-25T18:27:28.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018680152 0 0) 0x7fa4a4135260 con 0x7fa4a41430c0 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a4137340 con 0x7fa4a41430c0 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa49c001bd0 con 0x7fa4a41430c0 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:28.908 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:28.909 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:28.909 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.907+0000 7fa4a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038679629 0 0) 0x7fa490004eb0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a4135260 con 0x7fa4a40a98f0 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa494001970 con 0x7fa4a40a98f0 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.915+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:28.916 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:28.917 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410328370 0 0) 0x7fa4a4135fe0 con 0x7fa4a413f710 2024-03-25T18:27:28.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa490004eb0 con 0x7fa4a413f710 2024-03-25T18:27:28.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa498001b00 con 0x7fa4a413f710 2024-03-25T18:27:28.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:28.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.923+0000 7fa4a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:28.925 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 424492 0 0) 0x7fa4a4135260 con 0x7fa4a40a98f0 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 >> v1:172.21.15.186:6789/0 conn(0x7fa4a413f710 legacy=0x7fa4a41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 >> v1:172.21.15.176:6789/0 conn(0x7fa4a41430c0 legacy=0x7fa4a41d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 1 monclient: found mon.c 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a41d5000 con 0x7fa4a40a98f0 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:28.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4aa967640 5 monclient: authenticate success, global_id 4373 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4aa967640 10 monclient: _renew_subs 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4aa967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.929+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a41d52b0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 10 monclient: _renew_subs 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41d43c0 con 0x7fa4a40a98f0 2024-03-25T18:27:28.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 1 librados: init done 2024-03-25T18:27:28.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.930+0000 7fa4aa967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:28.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.931+0000 7fa4aa967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:28.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.931+0000 7fa4aa967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:28.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:28.931+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a40627b0 con 0x7fa4a40a98f0 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4940035f0 con 0x7fa4a40a98f0 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa494003900 con 0x7fa4a40a98f0 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a0ff9640 10 monclient: dump: 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:29.265 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:29.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.264+0000 7fa4a27fc640 4 set_mon_vals no callback set 2024-03-25T18:27:29.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.265+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa4940198d0 con 0x7fa4a40a98f0 2024-03-25T18:27:29.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.265+0000 7fa4a37fe640 10 monclient: get_auth_request con 0x7fa49006e130 auth_method 0 2024-03-25T18:27:29.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.265+0000 7fa4a0ff9640 10 client.4373.objecter ms_handle_connect 0x7fa49006e130 2024-03-25T18:27:29.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fa494053580 con 0x7fa4a40a98f0 2024-03-25T18:27:29.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 10 client.4373.objecter ms_dispatch 0x7fa4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa49404f090 con 0x7fa4a40a98f0 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:29.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.270+0000 7fa4a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:29.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.537+0000 7fa4aa967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.537+0000 7fa4aa967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.537+0000 7fa4aa967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:27:29.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.537+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4a4133760 con 0x7fa4a40a98f0 2024-03-25T18:27:29.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.576+0000 7fa4a0ff9640 1 -- 172.21.15.176:0/1490074560 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fa494050fc0 con 0x7fa4a40a98f0 2024-03-25T18:27:29.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.576+0000 7fa4a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:29.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.576+0000 7fa4a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:29.577 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:29.578 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 10 librados: watch_flush enter 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 10 librados: watch_flush exit 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 10 client.4373.objecter successfully canceled tick 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa49006e130 legacy=0x7fa4900705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 10 monclient: shutdown 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 >> v1:172.21.15.176:6790/0 conn(0x7fa4a40a98f0 legacy=0x7fa4a41358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:29.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 shutdown_connections 2024-03-25T18:27:29.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 >> 172.21.15.176:0/1490074560 conn(0x7fa4a40b7ea0 msgr2=0x7fa4a413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:29.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 shutdown_connections 2024-03-25T18:27:29.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.579+0000 7fa4aa967640 1 -- 172.21.15.176:0/1490074560 wait complete. 2024-03-25T18:27:29.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:29.580+0000 7fa4aa967640 1 librados: shutdown 2024-03-25T18:27:29.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:36.400 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:36.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.510+0000 7f7ce0225640 10 monclient: get_monmap_and_config 2024-03-25T18:27:36.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.510+0000 7f7ce0225640 10 monclient: build_initial_monmap 2024-03-25T18:27:36.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.510+0000 7f7ce0225640 10 monclient: monmap: 2024-03-25T18:27:36.511 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:36.511 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:36.511487+0000 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:36.511487+0000 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.511+0000 7f7ce0225640 1 Processor -- start 2024-03-25T18:27:36.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.511+0000 7f7ce0225640 1 -- start start 2024-03-25T18:27:36.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.511+0000 7f7ce0225640 10 monclient: init 2024-03-25T18:27:36.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:36.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:36.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd80a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd81430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8142520 con 0x7f7cd80a98f0 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b28a0 con 0x7f7cd813f710 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b2a30 con 0x7f7cd81430c0 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:36.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.512+0000 7f7ce0225640 10 monclient(hunting): authenticate will time out at 8334.824637s 2024-03-25T18:27:36.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.519+0000 7f7cddfc1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7cd80a98f0 0x7f7cd80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:39780/0 (socket says 172.21.15.176:39780) 2024-03-25T18:27:36.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.519+0000 7f7cddfc1640 1 -- 172.21.15.176:0/531642798 learned_addr learned my addr 172.21.15.176:0/531642798 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:36.540 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968709352 0 0) 0x7f7cd8142520 con 0x7f7cd80a98f0 2024-03-25T18:27:36.540 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:27:36.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.539+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4003520 con 0x7f7cd80a98f0 2024-03-25T18:27:36.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768699185 0 0) 0x7f7cd80b28a0 con 0x7f7cd813f710 2024-03-25T18:27:36.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:27:36.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.543+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd8142520 con 0x7f7cd813f710 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690612075 0 0) 0x7f7cd80b2a30 con 0x7f7cd81430c0 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:27:36.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.549+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd80b28a0 con 0x7f7cd81430c0 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652329973 0 0) 0x7f7cd8142520 con 0x7f7cd813f710 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 >> v1:172.21.15.176:6790/0 conn(0x7f7cd81430c0 legacy=0x7f7cd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 >> v1:172.21.15.176:6789/0 conn(0x7f7cd80a98f0 legacy=0x7f7cd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 10 monclient: _finish_hunting 0 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 monclient: found mon.noname-c 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:27:36.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80b44c0 con 0x7f7cd813f710 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 10 monclient: _finish_auth 0 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 10 monclient: _check_auth_tickets 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7ce0225640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.555+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7cc8001670 con 0x7f7cd813f710 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.556+0000 7f7cdcfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.556+0000 7f7cdcfbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.556+0000 7f7cdcfbf640 10 monclient: dump: 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.558 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.556+0000 7f7ce0225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cc80017d0 con 0x7f7cd813f710 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 1 -- 172.21.15.176:0/531642798 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7cc8001be0 con 0x7f7cd813f710 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7ce0225640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7ce0225640 4 set_mon_vals no callback set 2024-03-25T18:27:36.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.581+0000 7f7cdcfbf640 10 monclient: dump: 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.583 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.584 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:36.584 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:36.584 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:36.584 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 10 monclient: shutdown 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 1 -- 172.21.15.176:0/531642798 >> v1:172.21.15.186:6789/0 conn(0x7f7cd813f710 legacy=0x7f7cd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 1 -- 172.21.15.176:0/531642798 shutdown_connections 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 1 -- 172.21.15.176:0/531642798 >> 172.21.15.176:0/531642798 conn(0x7f7cd80b7ea0 msgr2=0x7f7cd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 1 -- 172.21.15.176:0/531642798 shutdown_connections 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.582+0000 7f7ce0225640 1 -- 172.21.15.176:0/531642798 wait complete. 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 10 monclient: build_initial_monmap 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 10 monclient: monmap: 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:36.586 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 librados: starting msgr at 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 librados: starting objecter 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 Processor -- start 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 -- start start 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 librados: setting wanted keys 2024-03-25T18:27:36.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 1 librados: calling monclient init 2024-03-25T18:27:36.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.583+0000 7f7ce0225640 10 monclient: init 2024-03-25T18:27:36.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:36.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd80a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd81430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd81d7a90 con 0x7f7cd81430c0 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b1700 con 0x7f7cd80a98f0 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b13d0 con 0x7f7cd813f710 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:36.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.584+0000 7f7ce0225640 10 monclient(hunting): authenticate will time out at 8334.896253s 2024-03-25T18:27:36.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.589+0000 7f7cddfc1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7cd80a98f0 0x7f7cd81d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:35834/0 (socket says 172.21.15.176:35834) 2024-03-25T18:27:36.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.589+0000 7f7cddfc1640 1 -- 172.21.15.176:0/4122854320 learned_addr learned my addr 172.21.15.176:0/4122854320 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:36.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.590+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd80a98f0 2024-03-25T18:27:36.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.590+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.591+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd813f710 2024-03-25T18:27:36.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.591+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126174947 0 0) 0x7f7cd80b1700 con 0x7f7cd80a98f0 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:27:36.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.602+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483835111 0 0) 0x7f7cd80b13d0 con 0x7f7cd813f710 2024-03-25T18:27:36.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:27:36.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.608+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd80b1700 con 0x7f7cd813f710 2024-03-25T18:27:36.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.622+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd81430c0 2024-03-25T18:27:36.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.622+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:36.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808111964 0 0) 0x7f7cbc004eb0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd80b13d0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7cd4001950 con 0x7f7cd80a98f0 2024-03-25T18:27:36.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.625 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.623+0000 7f7cce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:36.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721715047 0 0) 0x7f7cd81d7a90 con 0x7f7cd81430c0 2024-03-25T18:27:36.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:36.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:36.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:36.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:27:36.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.632+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd81430c0 2024-03-25T18:27:36.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232745421 0 0) 0x7f7cd80b1700 con 0x7f7cd813f710 2024-03-25T18:27:36.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd81d7a90 con 0x7f7cd813f710 2024-03-25T18:27:36.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7cc8001b20 con 0x7f7cd813f710 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.633+0000 7f7cce7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:36.635 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 105363166 0 0) 0x7f7cd80b13d0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 >> v1:172.21.15.176:6790/0 conn(0x7f7cd813f710 legacy=0x7f7cd81d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 >> v1:172.21.15.176:6789/0 conn(0x7f7cd81430c0 legacy=0x7f7cd814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 1 monclient: found mon.b 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:36.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd814eb50 con 0x7f7cd80a98f0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7cce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 10 monclient: _renew_subs 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd814dc50 con 0x7f7cd80a98f0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 10 monclient: _renew_subs 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.636+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd814e290 con 0x7f7cd80a98f0 2024-03-25T18:27:36.639 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.637+0000 7f7ce0225640 1 librados: init done 2024-03-25T18:27:36.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.637+0000 7f7ce0225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:36.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.637+0000 7f7ce0225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:36.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.637+0000 7f7ce0225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:36.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.637+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd80b1560 con 0x7f7cd80a98f0 2024-03-25T18:27:36.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd40047e0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:36.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7cd4004950 con 0x7f7cd80a98f0 2024-03-25T18:27:36.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7cce7fc640 10 monclient: dump: 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:36.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.943+0000 7f7ccf7fe640 4 set_mon_vals no callback set 2024-03-25T18:27:36.946 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.944+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f7cd401a920 con 0x7f7cd80a98f0 2024-03-25T18:27:36.946 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.945+0000 7f7cdd7c0640 10 monclient: get_auth_request con 0x7f7cbc06dfb0 auth_method 0 2024-03-25T18:27:36.946 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.945+0000 7f7cce7fc640 10 client.4366.objecter ms_handle_connect 0x7f7cbc06dfb0 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7cd40546b0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 10 client.4366.objecter ms_dispatch 0x7f7cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cd404fff0 con 0x7f7cd80a98f0 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:36.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:36.949+0000 7f7cce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:37.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.170+0000 7f7ce0225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.170+0000 7f7ce0225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.170+0000 7f7ce0225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:37.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.170+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd80b0d00 con 0x7f7cd80a98f0 2024-03-25T18:27:37.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.199+0000 7f7cce7fc640 1 -- 172.21.15.176:0/4122854320 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7cd40520f0 con 0x7f7cd80a98f0 2024-03-25T18:27:37.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.199+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:37.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.199+0000 7f7cce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:37.201 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:37.201 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:37.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.201+0000 7f7ce0225640 10 librados: watch_flush enter 2024-03-25T18:27:37.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 10 librados: watch_flush exit 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 10 client.4366.objecter successfully canceled tick 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 >> v1:172.21.15.176:6800/2598213983 conn(0x7f7cbc06dfb0 legacy=0x7f7cbc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 10 monclient: shutdown 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 >> v1:172.21.15.186:6789/0 conn(0x7f7cd80a98f0 legacy=0x7f7cd81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 shutdown_connections 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 >> 172.21.15.176:0/4122854320 conn(0x7f7cd80b7ea0 msgr2=0x7f7cd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 shutdown_connections 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 -- 172.21.15.176:0/4122854320 wait complete. 2024-03-25T18:27:37.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:37.202+0000 7f7ce0225640 1 librados: shutdown 2024-03-25T18:27:37.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:44.025 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:44.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.142+0000 7f0b605d0640 10 monclient: get_monmap_and_config 2024-03-25T18:27:44.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.142+0000 7f0b605d0640 10 monclient: build_initial_monmap 2024-03-25T18:27:44.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.142+0000 7f0b605d0640 10 monclient: monmap: 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:44.143504+0000 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:44.143504+0000 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:44.144 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.143+0000 7f0b605d0640 1 Processor -- start 2024-03-25T18:27:44.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.143+0000 7f0b605d0640 1 -- start start 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.143+0000 7f0b605d0640 10 monclient: init 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0b580a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0b5813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0b581430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b58142520 con 0x7f0b581430c0 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580b28a0 con 0x7f0b5813f710 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580b2a30 con 0x7f0b580a98f0 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:44.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.144+0000 7f0b605d0640 10 monclient(hunting): authenticate will time out at 8342.456265s 2024-03-25T18:27:44.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.153+0000 7f0b5eb6d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0b581430c0 0x7f0b5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:44536/0 (socket says 172.21.15.176:44536) 2024-03-25T18:27:44.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.153+0000 7f0b5eb6d640 1 -- 172.21.15.176:0/3442469563 learned_addr learned my addr 172.21.15.176:0/3442469563 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451747748 0 0) 0x7f0b580b28a0 con 0x7f0b5813f710 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:27:44.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.168+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44003520 con 0x7f0b5813f710 2024-03-25T18:27:44.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777548215 0 0) 0x7f0b58142520 con 0x7f0b581430c0 2024-03-25T18:27:44.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:27:44.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.175+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580b28a0 con 0x7f0b581430c0 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226836015 0 0) 0x7f0b580b2a30 con 0x7f0b580a98f0 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:27:44.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.178+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58142520 con 0x7f0b580a98f0 2024-03-25T18:27:44.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044282578 0 0) 0x7f0b44003520 con 0x7f0b5813f710 2024-03-25T18:27:44.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 >> v1:172.21.15.176:6790/0 conn(0x7f0b580a98f0 legacy=0x7f0b580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 >> v1:172.21.15.176:6789/0 conn(0x7f0b581430c0 legacy=0x7f0b5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 monclient: found mon.noname-c 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580b44c0 con 0x7f0b5813f710 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: _finish_auth 0 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b48001690 con 0x7f0b5813f710 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b605d0640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b605d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b5d36a640 10 monclient: dump: 2024-03-25T18:27:44.182 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.180+0000 7f0b605d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b48001870 con 0x7f0b5813f710 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 1 -- 172.21.15.176:0/3442469563 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b48001c80 con 0x7f0b5813f710 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b605d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b605d0640 4 set_mon_vals no callback set 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.189+0000 7f0b5d36a640 10 monclient: dump: 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.190 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:44.191 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 10 monclient: shutdown 2024-03-25T18:27:44.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:44.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 1 -- 172.21.15.176:0/3442469563 >> v1:172.21.15.186:6789/0 conn(0x7f0b5813f710 legacy=0x7f0b581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 1 -- 172.21.15.176:0/3442469563 shutdown_connections 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 1 -- 172.21.15.176:0/3442469563 >> 172.21.15.176:0/3442469563 conn(0x7f0b580b7ea0 msgr2=0x7f0b580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.190+0000 7f0b605d0640 1 -- 172.21.15.176:0/3442469563 shutdown_connections 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.191+0000 7f0b605d0640 1 -- 172.21.15.176:0/3442469563 wait complete. 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.191+0000 7f0b605d0640 10 monclient: build_initial_monmap 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.191+0000 7f0b605d0640 10 monclient: monmap: 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:44.193 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.191+0000 7f0b605d0640 1 librados: starting msgr at 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.191+0000 7f0b605d0640 1 librados: starting objecter 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 Processor -- start 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 -- start start 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 librados: setting wanted keys 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 librados: calling monclient init 2024-03-25T18:27:44.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient: init 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0b580a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0b581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580b72c0 con 0x7f0b5813f710 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5813b110 con 0x7f0b580a98f0 2024-03-25T18:27:44.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5813b380 con 0x7f0b581430c0 2024-03-25T18:27:44.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:44.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.192+0000 7f0b605d0640 10 monclient(hunting): authenticate will time out at 8342.504817s 2024-03-25T18:27:44.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.206+0000 7f0b5e36c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0b580a98f0 0x7f0b5813b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:40488/0 (socket says 172.21.15.176:40488) 2024-03-25T18:27:44.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.206+0000 7f0b5e36c640 1 -- 172.21.15.176:0/1941097564 learned_addr learned my addr 172.21.15.176:0/1941097564 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:44.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.209+0000 7f0b4effd640 10 client.?.objecter ms_handle_connect 0x7f0b580a98f0 2024-03-25T18:27:44.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.209+0000 7f0b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.209+0000 7f0b4effd640 10 client.?.objecter ms_handle_connect 0x7f0b5813f710 2024-03-25T18:27:44.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.209+0000 7f0b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.210+0000 7f0b4effd640 10 client.?.objecter ms_handle_connect 0x7f0b581430c0 2024-03-25T18:27:44.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.210+0000 7f0b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660718578 0 0) 0x7f0b580b44c0 con 0x7f0b5813f710 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:27:44.219 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.218+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c004eb0 con 0x7f0b5813f710 2024-03-25T18:27:44.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551022546 0 0) 0x7f0b5813b110 con 0x7f0b580a98f0 2024-03-25T18:27:44.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:27:44.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.228+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580b44c0 con 0x7f0b580a98f0 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576855142 0 0) 0x7f0b5813b380 con 0x7f0b581430c0 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:27:44.247 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.246+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5813b110 con 0x7f0b581430c0 2024-03-25T18:27:44.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.249+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942318254 0 0) 0x7f0b3c004eb0 con 0x7f0b5813f710 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.249+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b5813b380 con 0x7f0b5813f710 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.250+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b48001a20 con 0x7f0b5813f710 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.250+0000 7f0b4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.250+0000 7f0b4effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.250+0000 7f0b4effd640 10 monclient(hunting): dump: 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.251 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.250+0000 7f0b4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1335450481 0 0) 0x7f0b5813b380 con 0x7f0b5813f710 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 >> v1:172.21.15.176:6790/0 conn(0x7f0b581430c0 legacy=0x7f0b581e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 >> v1:172.21.15.186:6789/0 conn(0x7f0b580a98f0 legacy=0x7f0b5813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 1 monclient: found mon.a 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580b7970 con 0x7f0b5813f710 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 10 monclient: _finish_auth 0 2024-03-25T18:27:44.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b605d0640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b605d0640 10 monclient: _renew_subs 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b605d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.263+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58139810 con 0x7f0b5813f710 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.264+0000 7f0b605d0640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.264+0000 7f0b605d0640 10 monclient: _renew_subs 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.264+0000 7f0b605d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.264+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b58139f00 con 0x7f0b5813f710 2024-03-25T18:27:44.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.264+0000 7f0b605d0640 1 librados: init done 2024-03-25T18:27:44.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.265+0000 7f0b605d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.265+0000 7f0b605d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.265+0000 7f0b605d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:27:44.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.265+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b580b7bc0 con 0x7f0b5813f710 2024-03-25T18:27:44.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b48001c60 con 0x7f0b5813f710 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0b48003ae0 con 0x7f0b5813f710 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b4effd640 10 monclient: dump: 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:44.589 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:44.590 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:44.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.587+0000 7f0b5cb69640 4 set_mon_vals no callback set 2024-03-25T18:27:44.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.588+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f0b48019ab0 con 0x7f0b5813f710 2024-03-25T18:27:44.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.589+0000 7f0b5e36c640 10 monclient: get_auth_request con 0x7f0b3c06dda0 auth_method 0 2024-03-25T18:27:44.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.589+0000 7f0b4effd640 10 client.4368.objecter ms_handle_connect 0x7f0b3c06dda0 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f0b48053650 con 0x7f0b5813f710 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 10 client.4368.objecter ms_dispatch 0x7f0b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b4804f160 con 0x7f0b5813f710 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:44.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.595+0000 7f0b4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:44.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.810+0000 7f0b605d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.810+0000 7f0b605d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.810+0000 7f0b605d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:27:44.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.810+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b580b7480 con 0x7f0b5813f710 2024-03-25T18:27:44.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.833+0000 7f0b4effd640 1 -- 172.21.15.176:0/1941097564 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f0b580b7480 con 0x7f0b5813f710 2024-03-25T18:27:44.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.833+0000 7f0b4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:44.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.833+0000 7f0b4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:44.835 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:44.836 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:44.838 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 10 librados: watch_flush enter 2024-03-25T18:27:44.838 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 10 librados: watch_flush exit 2024-03-25T18:27:44.838 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:27:44.838 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:27:44.838 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 >> v1:172.21.15.176:6800/2598213983 conn(0x7f0b3c06dda0 legacy=0x7f0b3c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 10 monclient: shutdown 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 >> v1:172.21.15.176:6789/0 conn(0x7f0b5813f710 legacy=0x7f0b5812f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 shutdown_connections 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 >> 172.21.15.176:0/1941097564 conn(0x7f0b580b7ea0 msgr2=0x7f0b5813e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:44.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 shutdown_connections 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.835+0000 7f0b605d0640 1 -- 172.21.15.176:0/1941097564 wait complete. 2024-03-25T18:27:44.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:44.836+0000 7f0b605d0640 1 librados: shutdown 2024-03-25T18:27:44.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:47.907 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:49.663 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:47.914 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:49.670 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:47.991 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:03:49.747 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.031 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:49.787 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.038 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:49.773 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.039 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:49.796 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.048 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:49.782 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.064 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:03:49.820 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.078 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:49.812 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:51.656 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:51.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.767+0000 7f0f5ea25640 10 monclient: get_monmap_and_config 2024-03-25T18:27:51.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.767+0000 7f0f5ea25640 10 monclient: build_initial_monmap 2024-03-25T18:27:51.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.767+0000 7f0f5ea25640 10 monclient: monmap: 2024-03-25T18:27:51.769 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:51.769 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:51.768487+0000 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:51.768487+0000 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.768+0000 7f0f5ea25640 1 Processor -- start 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.768+0000 7f0f5ea25640 1 -- start start 2024-03-25T18:27:51.770 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.768+0000 7f0f5ea25640 10 monclient: init 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-a con 0x7f0f580a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-b con 0x7f0f5813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-c con 0x7f0f581430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:51.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f58142520 con 0x7f0f580a98f0 2024-03-25T18:27:51.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f580b28a0 con 0x7f0f581430c0 2024-03-25T18:27:51.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f580b2a30 con 0x7f0f5813f710 2024-03-25T18:27:51.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:51.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.769+0000 7f0f5ea25640 10 monclient(hunting): authenticate will time out at 8350.081541s 2024-03-25T18:27:51.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.776+0000 7f0f577fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f0f5813f710 0x7f0f581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48746/0 (socket says 172.21.15.176:48746) 2024-03-25T18:27:51.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.776+0000 7f0f577fe640 1 -- 172.21.15.176:0/3022896972 learned_addr learned my addr 172.21.15.176:0/3022896972 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.797+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670407260 0 0) 0x7f0f580b2a30 con 0x7f0f5813f710 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.797+0000 7f0f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.797+0000 7f0f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.797+0000 7f0f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.797+0000 7f0f56ffd640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:27:51.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.798+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f40003520 con 0x7f0f5813f710 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775805961 0 0) 0x7f0f58142520 con 0x7f0f580a98f0 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:27:51.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.800+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f580b2a30 con 0x7f0f580a98f0 2024-03-25T18:27:51.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595496361 0 0) 0x7f0f580b28a0 con 0x7f0f581430c0 2024-03-25T18:27:51.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.811 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:27:51.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.810+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f58142520 con 0x7f0f581430c0 2024-03-25T18:27:51.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526338122 0 0) 0x7f0f40003520 con 0x7f0f5813f710 2024-03-25T18:27:51.819 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 >> v1:172.21.15.186:6789/0 conn(0x7f0f581430c0 legacy=0x7f0f5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 >> v1:172.21.15.176:6789/0 conn(0x7f0f580a98f0 legacy=0x7f0f580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580b44c0 con 0x7f0f5813f710 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0f4c001690 con 0x7f0f5813f710 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f56ffd640 10 monclient: dump: 2024-03-25T18:27:51.820 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f5ea25640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:27:51.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.818+0000 7f0f5ea25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:51.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f4c0017f0 con 0x7f0f5813f710 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 1 -- 172.21.15.176:0/3022896972 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0f4c001c00 con 0x7f0f5813f710 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f5ea25640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f5ea25640 4 set_mon_vals no callback set 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.839+0000 7f0f56ffd640 10 monclient: dump: 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:51.841 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:51.842 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:51.842 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:51.842 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 10 monclient: shutdown 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 1 -- 172.21.15.176:0/3022896972 >> v1:172.21.15.176:6790/0 conn(0x7f0f5813f710 legacy=0x7f0f581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 1 -- 172.21.15.176:0/3022896972 shutdown_connections 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 1 -- 172.21.15.176:0/3022896972 >> 172.21.15.176:0/3022896972 conn(0x7f0f580b7ea0 msgr2=0x7f0f580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 1 -- 172.21.15.176:0/3022896972 shutdown_connections 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.840+0000 7f0f5ea25640 1 -- 172.21.15.176:0/3022896972 wait complete. 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.841+0000 7f0f5ea25640 10 monclient: build_initial_monmap 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.841+0000 7f0f5ea25640 10 monclient: monmap: 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:51.844 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.841+0000 7f0f5ea25640 1 librados: starting msgr at 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.841+0000 7f0f5ea25640 1 librados: starting objecter 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.841+0000 7f0f5ea25640 1 Processor -- start 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 1 -- start start 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 1 librados: setting wanted keys 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 1 librados: calling monclient init 2024-03-25T18:27:51.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 10 monclient: init 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.842+0000 7f0f5ea25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-b con 0x7f0f580a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-c con 0x7f0f5813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 10 monclient(hunting): picked mon.noname-a con 0x7f0f581430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f58137df0 con 0x7f0f581430c0 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f581337a0 con 0x7f0f580a98f0 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f58133930 con 0x7f0f5813f710 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:51.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.843+0000 7f0f5ea25640 10 monclient(hunting): authenticate will time out at 8350.155076s 2024-03-25T18:27:51.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.859+0000 7f0f5cfc2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0f581430c0 0x7f0f58133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57928/0 (socket says 172.21.15.176:57928) 2024-03-25T18:27:51.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.859+0000 7f0f5cfc2640 1 -- 172.21.15.176:0/2627734800 learned_addr learned my addr 172.21.15.176:0/2627734800 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:51.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.863+0000 7f0f54ff9640 10 client.?.objecter ms_handle_connect 0x7f0f580a98f0 2024-03-25T18:27:51.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.863+0000 7f0f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:51.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.864+0000 7f0f54ff9640 10 client.?.objecter ms_handle_connect 0x7f0f581430c0 2024-03-25T18:27:51.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.864+0000 7f0f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691731531 0 0) 0x7f0f581337a0 con 0x7f0f580a98f0 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:27:51.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.869+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f44004eb0 con 0x7f0f580a98f0 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.871+0000 7f0f54ff9640 10 client.?.objecter ms_handle_connect 0x7f0f5813f710 2024-03-25T18:27:51.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.871+0000 7f0f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:51.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101158066 0 0) 0x7f0f58137df0 con 0x7f0f581430c0 2024-03-25T18:27:51.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.877+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f581337a0 con 0x7f0f581430c0 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474074250 0 0) 0x7f0f58133930 con 0x7f0f5813f710 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:51.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:51.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:27:51.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.879+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f58137df0 con 0x7f0f5813f710 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391813552 0 0) 0x7f0f44004eb0 con 0x7f0f580a98f0 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f58133930 con 0x7f0f580a98f0 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0f48001970 con 0x7f0f580a98f0 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:51.885 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.884+0000 7f0f54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:51.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614369172 0 0) 0x7f0f581337a0 con 0x7f0f581430c0 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f44004eb0 con 0x7f0f581430c0 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0f50001c50 con 0x7f0f581430c0 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.888+0000 7f0f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:51.890 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:51.891 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:51.891 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:51.891 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:51.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2740589650 0 0) 0x7f0f58133930 con 0x7f0f580a98f0 2024-03-25T18:27:51.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 >> v1:172.21.15.176:6790/0 conn(0x7f0f5813f710 legacy=0x7f0f581376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 >> v1:172.21.15.176:6789/0 conn(0x7f0f581430c0 legacy=0x7f0f58133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:51.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 1 monclient: found mon.b 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f581367d0 con 0x7f0f580a98f0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 10 monclient: _renew_subs 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f581369e0 con 0x7f0f580a98f0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 10 monclient: _renew_subs 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:51.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5813b640 con 0x7f0f580a98f0 2024-03-25T18:27:51.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.891+0000 7f0f5ea25640 1 librados: init done 2024-03-25T18:27:51.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.892+0000 7f0f5ea25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:51.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.892+0000 7f0f5ea25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:51.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.892+0000 7f0f5ea25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:51.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:51.892+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f580627b0 con 0x7f0f580a98f0 2024-03-25T18:27:52.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f480035f0 con 0x7f0f580a98f0 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0f48003900 con 0x7f0f580a98f0 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f54ff9640 10 monclient: dump: 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:52.183 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:52.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.181+0000 7f0f567fc640 4 set_mon_vals no callback set 2024-03-25T18:27:52.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.182+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f0f48019910 con 0x7f0f580a98f0 2024-03-25T18:27:52.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.182+0000 7f0f577fe640 10 monclient: get_auth_request con 0x7f0f4406ded0 auth_method 0 2024-03-25T18:27:52.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.183+0000 7f0f54ff9640 10 client.4378.objecter ms_handle_connect 0x7f0f4406ded0 2024-03-25T18:27:52.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.186+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f0f48053680 con 0x7f0f580a98f0 2024-03-25T18:27:52.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.186+0000 7f0f54ff9640 10 client.4378.objecter ms_dispatch 0x7f0f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.186+0000 7f0f54ff9640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.186+0000 7f0f54ff9640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.187+0000 7f0f54ff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.187+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f4804efc0 con 0x7f0f580a98f0 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.187+0000 7f0f54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:52.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.187+0000 7f0f54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:52.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.418+0000 7f0f5ea25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.418+0000 7f0f5ea25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.418+0000 7f0f5ea25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:52.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.418+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f58074cc0 con 0x7f0f580a98f0 2024-03-25T18:27:52.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.479+0000 7f0f54ff9640 1 -- 172.21.15.176:0/2627734800 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f0f480510c0 con 0x7f0f580a98f0 2024-03-25T18:27:52.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.479+0000 7f0f54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:52.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.479+0000 7f0f54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:52.481 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:27:52.481 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:52.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.481+0000 7f0f5ea25640 10 librados: watch_flush enter 2024-03-25T18:27:52.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 10 librados: watch_flush exit 2024-03-25T18:27:52.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 >> v1:172.21.15.176:6800/2598213983 conn(0x7f0f4406ded0 legacy=0x7f0f44070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 10 monclient: shutdown 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 >> v1:172.21.15.186:6789/0 conn(0x7f0f580a98f0 legacy=0x7f0f581392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 shutdown_connections 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 >> 172.21.15.176:0/2627734800 conn(0x7f0f580b7ea0 msgr2=0x7f0f58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 shutdown_connections 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 -- 172.21.15.176:0/2627734800 wait complete. 2024-03-25T18:27:52.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:52.482+0000 7f0f5ea25640 1 librados: shutdown 2024-03-25T18:27:52.495 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:59.303 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.420+0000 7fc6eadc6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.420+0000 7fc6eadc6640 10 monclient: build_initial_monmap 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.421+0000 7fc6eadc6640 10 monclient: monmap: 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:27:59.421838+0000 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:27:59.421838+0000 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:59.422 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.423 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:59.423 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:27:59.423 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:27:59.423 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 1 Processor -- start 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 1 -- start start 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient: init 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:59.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4142520 con 0x7fc6e40a98f0 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40b28a0 con 0x7fc6e41430c0 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e40b2a30 con 0x7fc6e413f710 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:59.425 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.422+0000 7fc6eadc6640 10 monclient(hunting): authenticate will time out at 8357.734812s 2024-03-25T18:27:59.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.430+0000 7fc6e3fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc6e413f710 0x7fc6e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:34798/0 (socket says 172.21.15.176:34798) 2024-03-25T18:27:59.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.430+0000 7fc6e3fff640 1 -- 172.21.15.176:0/3319183326 learned_addr learned my addr 172.21.15.176:0/3319183326 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:59.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419642773 0 0) 0x7fc6e40b28a0 con 0x7fc6e41430c0 2024-03-25T18:27:59.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:27:59.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.436+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d4003520 con 0x7fc6e41430c0 2024-03-25T18:27:59.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259295085 0 0) 0x7fc6e40b2a30 con 0x7fc6e413f710 2024-03-25T18:27:59.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:27:59.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.438+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e40b28a0 con 0x7fc6e413f710 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059455455 0 0) 0x7fc6d4003520 con 0x7fc6e41430c0 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 >> v1:172.21.15.176:6790/0 conn(0x7fc6e413f710 legacy=0x7fc6e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 >> v1:172.21.15.176:6789/0 conn(0x7fc6e40a98f0 legacy=0x7fc6e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 monclient: found mon.noname-c 2024-03-25T18:27:59.443 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e40b44c0 con 0x7fc6e41430c0 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6dc001a10 con 0x7fc6e41430c0 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6e37fe640 10 monclient: dump: 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.444 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.445 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.445 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.445 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6eadc6640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:27:59.445 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.442+0000 7fc6eadc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:59.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.447+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6dc001b70 con 0x7fc6e41430c0 2024-03-25T18:27:59.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.447+0000 7fc6e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:59.448 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.447+0000 7fc6e37fe640 1 -- 172.21.15.176:0/3319183326 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6dc001f80 con 0x7fc6e41430c0 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6eadc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6eadc6640 4 set_mon_vals no callback set 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6e37fe640 10 monclient: dump: 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.449 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.450 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6eadc6640 10 monclient: shutdown 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6eadc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.448+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3319183326 >> v1:172.21.15.186:6789/0 conn(0x7fc6e41430c0 legacy=0x7fc6e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3319183326 shutdown_connections 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3319183326 >> 172.21.15.176:0/3319183326 conn(0x7fc6e40b7ea0 msgr2=0x7fc6e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3319183326 shutdown_connections 2024-03-25T18:27:59.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3319183326 wait complete. 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 10 monclient: build_initial_monmap 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.449+0000 7fc6eadc6640 10 monclient: monmap: 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 librados: starting msgr at 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 librados: starting objecter 2024-03-25T18:27:59.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 Processor -- start 2024-03-25T18:27:59.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 -- start start 2024-03-25T18:27:59.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 librados: setting wanted keys 2024-03-25T18:27:59.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 1 librados: calling monclient init 2024-03-25T18:27:59.453 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.450+0000 7fc6eadc6640 10 monclient: init 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e41d7500 con 0x7fc6e40a98f0 2024-03-25T18:27:59.454 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e41394a0 con 0x7fc6e41430c0 2024-03-25T18:27:59.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4138b50 con 0x7fc6e413f710 2024-03-25T18:27:59.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:59.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.451+0000 7fc6eadc6640 10 monclient(hunting): authenticate will time out at 8357.763565s 2024-03-25T18:27:59.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.461+0000 7fc6e3fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc6e413f710 0x7fc6e41d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:34808/0 (socket says 172.21.15.176:34808) 2024-03-25T18:27:59.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.461+0000 7fc6e3fff640 1 -- 172.21.15.176:0/3053226616 learned_addr learned my addr 172.21.15.176:0/3053226616 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:27:59.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.465+0000 7fc6e17fa640 10 client.?.objecter ms_handle_connect 0x7fc6e413f710 2024-03-25T18:27:59.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.465+0000 7fc6e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.472+0000 7fc6e17fa640 10 client.?.objecter ms_handle_connect 0x7fc6e41430c0 2024-03-25T18:27:59.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.472+0000 7fc6e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.473+0000 7fc6e17fa640 10 client.?.objecter ms_handle_connect 0x7fc6e40a98f0 2024-03-25T18:27:59.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.473+0000 7fc6e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:59.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172553626 0 0) 0x7fc6e4138b50 con 0x7fc6e413f710 2024-03-25T18:27:59.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:27:59.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.478+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c8004eb0 con 0x7fc6e413f710 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308047669 0 0) 0x7fc6e41394a0 con 0x7fc6e41430c0 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:27:59.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.481+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e4138b50 con 0x7fc6e41430c0 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039352652 0 0) 0x7fc6e41d7500 con 0x7fc6e40a98f0 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:27:59.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.490+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e41394a0 con 0x7fc6e40a98f0 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280560907 0 0) 0x7fc6c8004eb0 con 0x7fc6e413f710 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e41d7500 con 0x7fc6e413f710 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6d00019b0 con 0x7fc6e413f710 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.496 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.494+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670701438 0 0) 0x7fc6e4138b50 con 0x7fc6e41430c0 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.495+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c8004eb0 con 0x7fc6e41430c0 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.495+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6dc001e80 con 0x7fc6e41430c0 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.495+0000 7fc6e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.495+0000 7fc6e17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.495+0000 7fc6e17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.497 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.498 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.498 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.498 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.499+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3708965367 0 0) 0x7fc6e41394a0 con 0x7fc6e40a98f0 2024-03-25T18:27:59.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.499+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e4138b50 con 0x7fc6e40a98f0 2024-03-25T18:27:59.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.499+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6d8001860 con 0x7fc6e40a98f0 2024-03-25T18:27:59.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.499+0000 7fc6e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.500+0000 7fc6e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.500+0000 7fc6e17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.501 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3412895532 0 0) 0x7fc6c8004eb0 con 0x7fc6e41430c0 2024-03-25T18:27:59.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 >> v1:172.21.15.176:6790/0 conn(0x7fc6e413f710 legacy=0x7fc6e41d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 >> v1:172.21.15.176:6789/0 conn(0x7fc6e40a98f0 legacy=0x7fc6e4138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:59.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:59.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 1 monclient: found mon.b 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e41d8380 con 0x7fc6e41430c0 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 10 monclient: _renew_subs 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e4139710 con 0x7fc6e41430c0 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 10 monclient: _renew_subs 2024-03-25T18:27:59.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:59.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e41d7750 con 0x7fc6e41430c0 2024-03-25T18:27:59.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.507+0000 7fc6eadc6640 1 librados: init done 2024-03-25T18:27:59.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.508+0000 7fc6eadc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:59.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.508+0000 7fc6eadc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:59.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.508+0000 7fc6eadc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:27:59.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.508+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e40627b0 con 0x7fc6e41430c0 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6dc002040 con 0x7fc6e41430c0 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:59.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc6dc003dc0 con 0x7fc6e41430c0 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 10 monclient: dump: 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:27:59.781 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:27:59.782 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:27:59.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e2ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:59.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.779+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fc6dc019d90 con 0x7fc6e41430c0 2024-03-25T18:27:59.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.780+0000 7fc6e8b62640 10 monclient: get_auth_request con 0x7fc6c806e160 auth_method 0 2024-03-25T18:27:59.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.780+0000 7fc6e17fa640 10 client.4384.objecter ms_handle_connect 0x7fc6c806e160 2024-03-25T18:27:59.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fc6dc053920 con 0x7fc6e41430c0 2024-03-25T18:27:59.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 10 client.4384.objecter ms_dispatch 0x7fc6e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:59.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.784+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6dc04f430 con 0x7fc6e41430c0 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.785+0000 7fc6e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:59.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:27:59.785+0000 7fc6e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:00.047 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.046+0000 7fc6eadc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.046+0000 7fc6eadc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.046+0000 7fc6eadc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:00.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.046+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e4069f40 con 0x7fc6e41430c0 2024-03-25T18:28:00.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.067+0000 7fc6e17fa640 1 -- 172.21.15.176:0/3053226616 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fc6dc051360 con 0x7fc6e41430c0 2024-03-25T18:28:00.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.067+0000 7fc6e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:00.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.067+0000 7fc6e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:00.068 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:00.068 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:00.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 10 librados: watch_flush enter 2024-03-25T18:28:00.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 10 librados: watch_flush exit 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 >> v1:172.21.15.176:6800/2598213983 conn(0x7fc6c806e160 legacy=0x7fc6c8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 10 monclient: shutdown 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 >> v1:172.21.15.186:6789/0 conn(0x7fc6e41430c0 legacy=0x7fc6e4134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 shutdown_connections 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 >> 172.21.15.176:0/3053226616 conn(0x7fc6e40b7ea0 msgr2=0x7fc6e413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 shutdown_connections 2024-03-25T18:28:00.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 -- 172.21.15.176:0/3053226616 wait complete. 2024-03-25T18:28:00.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:00.069+0000 7fc6eadc6640 1 librados: shutdown 2024-03-25T18:28:00.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:06.889 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:07.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.004+0000 7f53d9684640 10 monclient: get_monmap_and_config 2024-03-25T18:28:07.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.004+0000 7f53d9684640 10 monclient: build_initial_monmap 2024-03-25T18:28:07.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.004+0000 7f53d9684640 10 monclient: monmap: 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:07.005541+0000 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:07.005541+0000 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:07.006 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 1 Processor -- start 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 1 -- start start 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 10 monclient: init 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.005+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-a con 0x7f53d40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-b con 0x7f53d413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-c con 0x7f53d41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d4142520 con 0x7f53d40a98f0 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d40b28a0 con 0x7f53d41430c0 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d40b2a30 con 0x7f53d413f710 2024-03-25T18:28:07.008 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:07.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.006+0000 7f53d9684640 10 monclient(hunting): authenticate will time out at 8365.317988s 2024-03-25T18:28:07.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.015+0000 7f53d2ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f53d40a98f0 0x7f53d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:38658/0 (socket says 172.21.15.176:38658) 2024-03-25T18:28:07.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.015+0000 7f53d2ffd640 1 -- 172.21.15.176:0/2950118473 learned_addr learned my addr 172.21.15.176:0/2950118473 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502092891 0 0) 0x7f53d40b28a0 con 0x7f53d41430c0 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:28:07.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.022+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53bc003520 con 0x7f53d41430c0 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132222863 0 0) 0x7f53d40b2a30 con 0x7f53d413f710 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:28:07.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.023+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d40b28a0 con 0x7f53d413f710 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452390016 0 0) 0x7f53d4142520 con 0x7f53d40a98f0 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:28:07.030 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.029+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d40b2a30 con 0x7f53d40a98f0 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001869708 0 0) 0x7f53d40b28a0 con 0x7f53d413f710 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 >> v1:172.21.15.186:6789/0 conn(0x7f53d41430c0 legacy=0x7f53d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 >> v1:172.21.15.176:6789/0 conn(0x7f53d40a98f0 legacy=0x7f53d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:28:07.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40b44c0 con 0x7f53d413f710 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f53c8001670 con 0x7f53d413f710 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d1ffb640 10 monclient: dump: 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:07.037 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:07.038 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:07.038 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d9684640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:28:07.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.035+0000 7f53d9684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:07.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c80017d0 con 0x7f53d413f710 2024-03-25T18:28:07.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:07.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 1 -- 172.21.15.176:0/2950118473 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f53c8001be0 con 0x7f53d413f710 2024-03-25T18:28:07.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d1ffb640 10 monclient: dump: 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d9684640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d9684640 4 set_mon_vals no callback set 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d9684640 10 monclient: shutdown 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d9684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.051+0000 7f53d9684640 1 -- 172.21.15.176:0/2950118473 >> v1:172.21.15.176:6790/0 conn(0x7f53d413f710 legacy=0x7f53d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.052+0000 7f53d9684640 1 -- 172.21.15.176:0/2950118473 shutdown_connections 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.052+0000 7f53d9684640 1 -- 172.21.15.176:0/2950118473 >> 172.21.15.176:0/2950118473 conn(0x7f53d40b7ea0 msgr2=0x7f53d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.052+0000 7f53d9684640 1 -- 172.21.15.176:0/2950118473 shutdown_connections 2024-03-25T18:28:07.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.052+0000 7f53d9684640 1 -- 172.21.15.176:0/2950118473 wait complete. 2024-03-25T18:28:07.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.053+0000 7f53d9684640 10 monclient: build_initial_monmap 2024-03-25T18:28:07.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.053+0000 7f53d9684640 10 monclient: monmap: 2024-03-25T18:28:07.057 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:07.057 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:07.057 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.053+0000 7f53d9684640 1 librados: starting msgr at 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.053+0000 7f53d9684640 1 librados: starting objecter 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.054+0000 7f53d9684640 1 Processor -- start 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.054+0000 7f53d9684640 1 -- start start 2024-03-25T18:28:07.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.054+0000 7f53d9684640 1 librados: setting wanted keys 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.054+0000 7f53d9684640 1 librados: calling monclient init 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.054+0000 7f53d9684640 10 monclient: init 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-a con 0x7f53d40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-c con 0x7f53d413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 10 monclient(hunting): picked mon.noname-b con 0x7f53d41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d4136fa0 con 0x7f53d40a98f0 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.055+0000 7f53d9684640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d413b780 con 0x7f53d41430c0 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.056+0000 7f53d9684640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d41d7ba0 con 0x7f53d413f710 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.056+0000 7f53d9684640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:07.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.056+0000 7f53d9684640 10 monclient(hunting): authenticate will time out at 8365.368040s 2024-03-25T18:28:07.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.060+0000 7f53d37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f53d41430c0 0x7f53d41e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:35744/0 (socket says 172.21.15.176:35744) 2024-03-25T18:28:07.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.060+0000 7f53d37fe640 1 -- 172.21.15.176:0/3803306287 learned_addr learned my addr 172.21.15.176:0/3803306287 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:07.063 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.063+0000 7f53bb7fe640 10 client.?.objecter ms_handle_connect 0x7f53d41430c0 2024-03-25T18:28:07.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.063+0000 7f53bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813240413 0 0) 0x7f53d413b780 con 0x7f53d41430c0 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:28:07.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.070+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c0004eb0 con 0x7f53d41430c0 2024-03-25T18:28:07.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.072+0000 7f53bb7fe640 10 client.?.objecter ms_handle_connect 0x7f53d413f710 2024-03-25T18:28:07.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.072+0000 7f53bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.073+0000 7f53bb7fe640 10 client.?.objecter ms_handle_connect 0x7f53d40a98f0 2024-03-25T18:28:07.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.073+0000 7f53bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194715275 0 0) 0x7f53d4136fa0 con 0x7f53d40a98f0 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:28:07.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.077+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d413b780 con 0x7f53d40a98f0 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.078+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404745988 0 0) 0x7f53c0004eb0 con 0x7f53d41430c0 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d4136fa0 con 0x7f53d41430c0 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f53cc001c30 con 0x7f53d41430c0 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:07.080 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:07.081 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:07.081 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.079+0000 7f53bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:07.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3741710152 0 0) 0x7f53d413b780 con 0x7f53d40a98f0 2024-03-25T18:28:07.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c0004eb0 con 0x7f53d40a98f0 2024-03-25T18:28:07.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f53c4001870 con 0x7f53d40a98f0 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.092+0000 7f53bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:07.094 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3893283306 0 0) 0x7f53d4136fa0 con 0x7f53d41430c0 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 >> v1:172.21.15.176:6790/0 conn(0x7f53d413f710 legacy=0x7f53d41d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 >> v1:172.21.15.176:6789/0 conn(0x7f53d40a98f0 legacy=0x7f53d4136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 1 monclient: found mon.b 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d4137990 con 0x7f53d41430c0 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:07.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53d9684640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53d9684640 10 monclient: _renew_subs 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53d9684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.098+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4137c40 con 0x7f53d41430c0 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.099+0000 7f53d9684640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.099+0000 7f53d9684640 10 monclient: _renew_subs 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.099+0000 7f53d9684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.099+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d4139860 con 0x7f53d41430c0 2024-03-25T18:28:07.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.099+0000 7f53d9684640 1 librados: init done 2024-03-25T18:28:07.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.100+0000 7f53d9684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.100+0000 7f53d9684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.100+0000 7f53d9684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:07.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.100+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d4137640 con 0x7f53d41430c0 2024-03-25T18:28:07.366 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53cc0020e0 con 0x7f53d41430c0 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f53cc003990 con 0x7f53d41430c0 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53bb7fe640 10 monclient: dump: 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:07.367 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.365+0000 7f53d17fa640 4 set_mon_vals no callback set 2024-03-25T18:28:07.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.366+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f53cc003b70 con 0x7f53d41430c0 2024-03-25T18:28:07.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.367+0000 7f53d2ffd640 10 monclient: get_auth_request con 0x7f53c006de90 auth_method 0 2024-03-25T18:28:07.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.367+0000 7f53bb7fe640 10 client.4390.objecter ms_handle_connect 0x7f53c006de90 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.374+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f53cc054900 con 0x7f53d41430c0 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.374+0000 7f53bb7fe640 10 client.4390.objecter ms_dispatch 0x7f53d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.374+0000 7f53bb7fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.374+0000 7f53bb7fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.374+0000 7f53bb7fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T18:28:07.375 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.375+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53cc050240 con 0x7f53d41430c0 2024-03-25T18:28:07.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.375+0000 7f53bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:07.376 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.375+0000 7f53bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:07.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.604+0000 7f53d9684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:07.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.604+0000 7f53d9684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:07.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.604+0000 7f53d9684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:07.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.604+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53d40627b0 con 0x7f53d41430c0 2024-03-25T18:28:07.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.623+0000 7f53bb7fe640 1 -- 172.21.15.176:0/3803306287 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f53cc052340 con 0x7f53d41430c0 2024-03-25T18:28:07.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.623+0000 7f53bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:07.624 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.623+0000 7f53bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:07.624 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:07.624 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 10 librados: watch_flush enter 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 10 librados: watch_flush exit 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 10 client.4390.objecter successfully canceled tick 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 >> v1:172.21.15.176:6800/2598213983 conn(0x7f53c006de90 legacy=0x7f53c0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 10 monclient: shutdown 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 >> v1:172.21.15.186:6789/0 conn(0x7f53d41430c0 legacy=0x7f53d41e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 shutdown_connections 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 >> 172.21.15.176:0/3803306287 conn(0x7f53d40b7ea0 msgr2=0x7f53d414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 shutdown_connections 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.625+0000 7f53d9684640 1 -- 172.21.15.176:0/3803306287 wait complete. 2024-03-25T18:28:07.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:07.626+0000 7f53d9684640 1 librados: shutdown 2024-03-25T18:28:07.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:07.994 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:09.750 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.041 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:09.775 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.043 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:09.799 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.045 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:09.780 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.066 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:09.822 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.076 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:09.810 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.105 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:09.861 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.111 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:09.867 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.148 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:09.904 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:14.446 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:14.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.560+0000 7f08f870e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:14.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.560+0000 7f08f870e640 10 monclient: build_initial_monmap 2024-03-25T18:28:14.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.560+0000 7f08f870e640 10 monclient: monmap: 2024-03-25T18:28:14.561 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:14.561 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:14.561435+0000 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:14.561435+0000 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:14.562 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.561+0000 7f08f870e640 1 Processor -- start 2024-03-25T18:28:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.561+0000 7f08f870e640 1 -- start start 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.561+0000 7f08f870e640 10 monclient: init 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-c con 0x7f08f00b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-b con 0x7f08f00b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-a con 0x7f08f014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f00af940 con 0x7f08f014be90 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f00b0a70 con 0x7f08f00b4d60 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f00b0d50 con 0x7f08f00b1730 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.562+0000 7f08f870e640 10 monclient(hunting): authenticate will time out at 8372.874420s 2024-03-25T18:28:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.564+0000 7f08f64aa640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f08f00b4d60 0x7f08f013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:41842/0 (socket says 172.21.15.176:41842) 2024-03-25T18:28:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.564+0000 7f08f64aa640 1 -- 172.21.15.176:0/618905389 learned_addr learned my addr 172.21.15.176:0/618905389 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:14.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532744831 0 0) 0x7f08f00b0a70 con 0x7f08f00b4d60 2024-03-25T18:28:14.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:28:14.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.573+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc003520 con 0x7f08f00b4d60 2024-03-25T18:28:14.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448935344 0 0) 0x7f08f00b0d50 con 0x7f08f00b1730 2024-03-25T18:28:14.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:28:14.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.598+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f00b0a70 con 0x7f08f00b1730 2024-03-25T18:28:14.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176287368 0 0) 0x7f08f00b0a70 con 0x7f08f00b1730 2024-03-25T18:28:14.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 >> v1:172.21.15.186:6789/0 conn(0x7f08f00b4d60 legacy=0x7f08f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 >> v1:172.21.15.176:6789/0 conn(0x7f08f014be90 legacy=0x7f08f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: _finish_hunting 0 2024-03-25T18:28:14.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 monclient: found mon.noname-b 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00b1030 con 0x7f08f00b1730 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: _finish_auth 0 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: _check_auth_tickets 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f08e0001670 con 0x7f08f00b1730 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f54a8640 10 monclient: dump: 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.609 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f870e640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:28:14.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.607+0000 7f08f870e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:14.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08e00017d0 con 0x7f08f00b1730 2024-03-25T18:28:14.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:14.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 1 -- 172.21.15.176:0/618905389 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f08e0001be0 con 0x7f08f00b1730 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f54a8640 10 monclient: dump: 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:14.630 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:14.631 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f870e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:14.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.625+0000 7f08f870e640 4 set_mon_vals no callback set 2024-03-25T18:28:14.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.626+0000 7f08f870e640 10 monclient: shutdown 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.626+0000 7f08f870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.626+0000 7f08f870e640 1 -- 172.21.15.176:0/618905389 >> v1:172.21.15.176:6790/0 conn(0x7f08f00b1730 legacy=0x7f08f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.626+0000 7f08f870e640 1 -- 172.21.15.176:0/618905389 shutdown_connections 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.626+0000 7f08f870e640 1 -- 172.21.15.176:0/618905389 >> 172.21.15.176:0/618905389 conn(0x7f08f00617a0 msgr2=0x7f08f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.627+0000 7f08f870e640 1 -- 172.21.15.176:0/618905389 shutdown_connections 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.627+0000 7f08f870e640 1 -- 172.21.15.176:0/618905389 wait complete. 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.627+0000 7f08f870e640 10 monclient: build_initial_monmap 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.627+0000 7f08f870e640 10 monclient: monmap: 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:14.633 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 librados: starting msgr at 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 librados: starting objecter 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 Processor -- start 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 -- start start 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 librados: setting wanted keys 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 1 librados: calling monclient init 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.628+0000 7f08f870e640 10 monclient: init 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:14.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-b con 0x7f08f00b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-c con 0x7f08f00b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient(hunting): picked mon.noname-a con 0x7f08f014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f0139550 con 0x7f08f014be90 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f0137b40 con 0x7f08f00b1730 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f0137cd0 con 0x7f08f00b4d60 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:14.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.629+0000 7f08f870e640 10 monclient(hunting): authenticate will time out at 8372.941133s 2024-03-25T18:28:14.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.645+0000 7f08f6cab640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f08f014be90 0x7f08f01e22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54754/0 (socket says 172.21.15.176:54754) 2024-03-25T18:28:14.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.645+0000 7f08f6cab640 1 -- 172.21.15.176:0/231889702 learned_addr learned my addr 172.21.15.176:0/231889702 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:14.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.649+0000 7f08e6ffd640 10 client.?.objecter ms_handle_connect 0x7f08f014be90 2024-03-25T18:28:14.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.649+0000 7f08e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.655 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.655+0000 7f08e6ffd640 10 client.?.objecter ms_handle_connect 0x7f08f00b1730 2024-03-25T18:28:14.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.655+0000 7f08e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.657+0000 7f08e6ffd640 10 client.?.objecter ms_handle_connect 0x7f08f00b4d60 2024-03-25T18:28:14.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.657+0000 7f08e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170246896 0 0) 0x7f08f0139550 con 0x7f08f014be90 2024-03-25T18:28:14.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.660 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:28:14.661 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.659+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d4004eb0 con 0x7f08f014be90 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745494045 0 0) 0x7f08f0137b40 con 0x7f08f00b1730 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:28:14.668 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.667+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0139550 con 0x7f08f00b1730 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894060844 0 0) 0x7f08f0137cd0 con 0x7f08f00b4d60 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:28:14.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.672+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0137b40 con 0x7f08f00b4d60 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708512961 0 0) 0x7f08d4004eb0 con 0x7f08f014be90 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08f0137cd0 con 0x7f08f014be90 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f08e8001c90 con 0x7f08f014be90 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.674 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.673+0000 7f08e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179162635 0 0) 0x7f08f0139550 con 0x7f08f00b1730 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d4004eb0 con 0x7f08f00b1730 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f08ec001950 con 0x7f08f00b1730 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.684+0000 7f08e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.685 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:14.686 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:14.686 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:14.686 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 451333236 0 0) 0x7f08f0137cd0 con 0x7f08f014be90 2024-03-25T18:28:14.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 >> v1:172.21.15.176:6790/0 conn(0x7f08f00b4d60 legacy=0x7f08f01db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 >> v1:172.21.15.186:6789/0 conn(0x7f08f00b1730 legacy=0x7f08f01dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 1 monclient: found mon.a 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f0144f80 con 0x7f08f014be90 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 10 monclient: _renew_subs 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f0145230 con 0x7f08f014be90 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:14.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 10 monclient: _renew_subs 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f0138fd0 con 0x7f08f014be90 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.690+0000 7f08f870e640 1 librados: init done 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.691+0000 7f08f870e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.691+0000 7f08f870e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.691+0000 7f08f870e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:14.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.691+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f013aa40 con 0x7f08f014be90 2024-03-25T18:28:14.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08e8003990 con 0x7f08f014be90 2024-03-25T18:28:14.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:14.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f08e8003d60 con 0x7f08f014be90 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08e6ffd640 10 monclient: dump: 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:14.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.994+0000 7f08f4ca7640 4 set_mon_vals no callback set 2024-03-25T18:28:14.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.995+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f08e8003f80 con 0x7f08f014be90 2024-03-25T18:28:14.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.996+0000 7f08f64aa640 10 monclient: get_auth_request con 0x7f08d406df70 auth_method 0 2024-03-25T18:28:14.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:14.996+0000 7f08e6ffd640 10 client.4392.objecter ms_handle_connect 0x7f08d406df70 2024-03-25T18:28:15.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f08e8054aa0 con 0x7f08f014be90 2024-03-25T18:28:15.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 10 client.4392.objecter ms_dispatch 0x7f08f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:15.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:15.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:15.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T18:28:15.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08e80505b0 con 0x7f08f014be90 2024-03-25T18:28:15.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.001+0000 7f08e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:15.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.215+0000 7f08f870e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.215+0000 7f08f870e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.215+0000 7f08f870e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:15.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.215+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08f00627b0 con 0x7f08f014be90 2024-03-25T18:28:15.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.256+0000 7f08e6ffd640 1 -- 172.21.15.176:0/231889702 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f08e80524e0 con 0x7f08f014be90 2024-03-25T18:28:15.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.256+0000 7f08e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:15.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.256+0000 7f08e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:15.258 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:15.258 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 10 librados: watch_flush enter 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 10 librados: watch_flush exit 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 10 client.4392.objecter successfully canceled tick 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 >> v1:172.21.15.176:6800/2598213983 conn(0x7f08d406df70 legacy=0x7f08d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 10 monclient: shutdown 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 >> v1:172.21.15.176:6789/0 conn(0x7f08f014be90 legacy=0x7f08f01e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 shutdown_connections 2024-03-25T18:28:15.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.258+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 >> 172.21.15.176:0/231889702 conn(0x7f08f00617a0 msgr2=0x7f08f00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.259+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 shutdown_connections 2024-03-25T18:28:15.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.259+0000 7f08f870e640 1 -- 172.21.15.176:0/231889702 wait complete. 2024-03-25T18:28:15.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:15.259+0000 7f08f870e640 1 librados: shutdown 2024-03-25T18:28:15.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:22.078 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.196+0000 7f467ff0e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.196+0000 7f467ff0e640 10 monclient: build_initial_monmap 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.196+0000 7f467ff0e640 10 monclient: monmap: 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:22.197259+0000 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:22.197259+0000 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:22.198 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:22.199 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.197+0000 7f467ff0e640 1 Processor -- start 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.197+0000 7f467ff0e640 1 -- start start 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.197+0000 7f467ff0e640 10 monclient: init 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-c con 0x7f46780b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7f46780b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-a con 0x7f467814be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780af940 con 0x7f467814be90 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780b0a70 con 0x7f46780b4d60 2024-03-25T18:28:22.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780b0d50 con 0x7f46780b1730 2024-03-25T18:28:22.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:22.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.198+0000 7f467ff0e640 10 monclient(hunting): authenticate will time out at 8380.510201s 2024-03-25T18:28:22.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.207+0000 7f467dcaa640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f46780b4d60 0x7f467813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:46578/0 (socket says 172.21.15.176:46578) 2024-03-25T18:28:22.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.207+0000 7f467dcaa640 1 -- 172.21.15.176:0/3235833696 learned_addr learned my addr 172.21.15.176:0/3235833696 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346048208 0 0) 0x7f46780b0a70 con 0x7f46780b4d60 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:28:22.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.214+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003520 con 0x7f46780b4d60 2024-03-25T18:28:22.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887930468 0 0) 0x7f46780b0d50 con 0x7f46780b1730 2024-03-25T18:28:22.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:28:22.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.220+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780b0a70 con 0x7f46780b1730 2024-03-25T18:28:22.222 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393601240 0 0) 0x7f4664003520 con 0x7f46780b4d60 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 >> v1:172.21.15.176:6790/0 conn(0x7f46780b1730 legacy=0x7f46780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 >> v1:172.21.15.176:6789/0 conn(0x7f467814be90 legacy=0x7f467814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: _finish_hunting 0 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 monclient: found mon.noname-c 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780b1030 con 0x7f46780b4d60 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: _finish_auth 0 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: _check_auth_tickets 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4674001a10 con 0x7f46780b4d60 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467cca8640 10 monclient: dump: 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467ff0e640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:28:22.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.222+0000 7f467ff0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:22.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.228+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674001b70 con 0x7f46780b4d60 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467cca8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467cca8640 1 -- 172.21.15.176:0/3235833696 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4674001f80 con 0x7f46780b4d60 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467cca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467cca8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467cca8640 10 monclient: dump: 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.230 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.231 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.231 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467ff0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:22.231 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467ff0e640 4 set_mon_vals no callback set 2024-03-25T18:28:22.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467ff0e640 10 monclient: shutdown 2024-03-25T18:28:22.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:22.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.229+0000 7f467ff0e640 1 -- 172.21.15.176:0/3235833696 >> v1:172.21.15.186:6789/0 conn(0x7f46780b4d60 legacy=0x7f467813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.230+0000 7f467ff0e640 1 -- 172.21.15.176:0/3235833696 shutdown_connections 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.230+0000 7f467ff0e640 1 -- 172.21.15.176:0/3235833696 >> 172.21.15.176:0/3235833696 conn(0x7f46780617a0 msgr2=0x7f4678061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.230+0000 7f467ff0e640 1 -- 172.21.15.176:0/3235833696 shutdown_connections 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.230+0000 7f467ff0e640 1 -- 172.21.15.176:0/3235833696 wait complete. 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 10 monclient: build_initial_monmap 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 10 monclient: monmap: 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:22.234 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 librados: starting msgr at 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 librados: starting objecter 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 Processor -- start 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 -- start start 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 librados: setting wanted keys 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 1 librados: calling monclient init 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.231+0000 7f467ff0e640 10 monclient: init 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-a con 0x7f46780b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7f46780b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient(hunting): picked mon.noname-c con 0x7f467814be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:22.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46781d84d0 con 0x7f46780b1730 2024-03-25T18:28:22.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678133c40 con 0x7f46780b4d60 2024-03-25T18:28:22.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678133f20 con 0x7f467814be90 2024-03-25T18:28:22.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:22.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.232+0000 7f467ff0e640 10 monclient(hunting): authenticate will time out at 8380.544465s 2024-03-25T18:28:22.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.240+0000 7f467dcaa640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f46780b1730 0x7f4678132be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59542/0 (socket says 172.21.15.176:59542) 2024-03-25T18:28:22.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.240+0000 7f467dcaa640 1 -- 172.21.15.176:0/2938533594 learned_addr learned my addr 172.21.15.176:0/2938533594 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:22.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.242+0000 7f466e7fc640 10 client.?.objecter ms_handle_connect 0x7f46780b1730 2024-03-25T18:28:22.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.242+0000 7f466e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.248+0000 7f466e7fc640 10 client.?.objecter ms_handle_connect 0x7f467814be90 2024-03-25T18:28:22.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.248+0000 7f466e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.251+0000 7f466e7fc640 10 client.?.objecter ms_handle_connect 0x7f46780b4d60 2024-03-25T18:28:22.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.251+0000 7f466e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:22.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090284498 0 0) 0x7f46781d84d0 con 0x7f46780b1730 2024-03-25T18:28:22.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:28:22.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.252+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c004eb0 con 0x7f46780b1730 2024-03-25T18:28:22.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982849709 0 0) 0x7f4678133c40 con 0x7f46780b4d60 2024-03-25T18:28:22.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46781d84d0 con 0x7f46780b4d60 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183624001 0 0) 0x7f4678133f20 con 0x7f467814be90 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:22.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:22.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:28:22.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.264+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678133c40 con 0x7f467814be90 2024-03-25T18:28:22.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2821001372 0 0) 0x7f46781d84d0 con 0x7f46780b4d60 2024-03-25T18:28:22.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4678133f20 con 0x7f46780b4d60 2024-03-25T18:28:22.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f46680018b0 con 0x7f46780b4d60 2024-03-25T18:28:22.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.278+0000 7f466e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 273505315 0 0) 0x7f4678133c40 con 0x7f467814be90 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46781d84d0 con 0x7f467814be90 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4670001690 con 0x7f467814be90 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.282+0000 7f466e7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.283 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.284 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4245750609 0 0) 0x7f465c004eb0 con 0x7f46780b1730 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4678133c40 con 0x7f46780b1730 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4674001f80 con 0x7f46780b1730 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.284+0000 7f466e7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.285 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.286 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.286 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.286 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.286 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.286 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 659361234 0 0) 0x7f4678133c40 con 0x7f46780b1730 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 >> v1:172.21.15.176:6790/0 conn(0x7f467814be90 legacy=0x7f46781351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 >> v1:172.21.15.186:6789/0 conn(0x7f46780b4d60 legacy=0x7f46781d7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 1 monclient: found mon.a 2024-03-25T18:28:22.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678134200 con 0x7f46780b1730 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f466e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 10 monclient: _renew_subs 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46781427e0 con 0x7f46780b1730 2024-03-25T18:28:22.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:22.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 10 monclient: _renew_subs 2024-03-25T18:28:22.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:22.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678135aa0 con 0x7f46780b1730 2024-03-25T18:28:22.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.292+0000 7f467ff0e640 1 librados: init done 2024-03-25T18:28:22.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.293+0000 7f467ff0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:22.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.293+0000 7f467ff0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:22.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.293+0000 7f467ff0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:22.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.293+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46781401f0 con 0x7f46780b1730 2024-03-25T18:28:22.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674004e10 con 0x7f46780b1730 2024-03-25T18:28:22.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:22.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f46740051a0 con 0x7f46780b1730 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466e7fc640 10 monclient: dump: 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:22.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.599+0000 7f466ffff640 4 set_mon_vals no callback set 2024-03-25T18:28:22.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.600+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f4674005380 con 0x7f46780b1730 2024-03-25T18:28:22.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.601+0000 7f467d4a9640 10 monclient: get_auth_request con 0x7f465c06e160 auth_method 0 2024-03-25T18:28:22.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.601+0000 7f466e7fc640 10 client.4395.objecter ms_handle_connect 0x7f465c06e160 2024-03-25T18:28:22.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f4674054d70 con 0x7f46780b1730 2024-03-25T18:28:22.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 10 client.4395.objecter ms_dispatch 0x7f4678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:22.621 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:22.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:22.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T18:28:22.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4674050880 con 0x7f46780b1730 2024-03-25T18:28:22.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:22.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.620+0000 7f466e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:22.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.833+0000 7f467ff0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:22.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.833+0000 7f467ff0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.833+0000 7f467ff0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:22.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.833+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46781d7a30 con 0x7f46780b1730 2024-03-25T18:28:22.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.892+0000 7f466e7fc640 1 -- 172.21.15.176:0/2938533594 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f46740527b0 con 0x7f46780b1730 2024-03-25T18:28:22.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.892+0000 7f466e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:22.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.892+0000 7f466e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:22.894 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:22.894 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 10 librados: watch_flush enter 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 10 librados: watch_flush exit 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 10 client.4395.objecter successfully canceled tick 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 >> v1:172.21.15.176:6800/2598213983 conn(0x7f465c06e160 legacy=0x7f465c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 10 monclient: shutdown 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 >> v1:172.21.15.176:6789/0 conn(0x7f46780b1730 legacy=0x7f4678132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 shutdown_connections 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 >> 172.21.15.176:0/2938533594 conn(0x7f46780617a0 msgr2=0x7f4678061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 shutdown_connections 2024-03-25T18:28:22.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 -- 172.21.15.176:0/2938533594 wait complete. 2024-03-25T18:28:22.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:22.895+0000 7f467ff0e640 1 librados: shutdown 2024-03-25T18:28:22.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:29.714 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:29.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.826+0000 7f7c2d63e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:29.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.826+0000 7f7c2d63e640 10 monclient: build_initial_monmap 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.826+0000 7f7c2d63e640 10 monclient: monmap: 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:29.827636+0000 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:29.827636+0000 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:29.828 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.827+0000 7f7c2d63e640 1 Processor -- start 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.827+0000 7f7c2d63e640 1 -- start start 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.827+0000 7f7c2d63e640 10 monclient: init 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-c con 0x7f7c2806ab80 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-a con 0x7f7c2806af60 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-b con 0x7f7c2813f990 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280afa60 con 0x7f7c2806af60 2024-03-25T18:28:29.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280afce0 con 0x7f7c2806ab80 2024-03-25T18:28:29.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280b1150 con 0x7f7c2813f990 2024-03-25T18:28:29.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:29.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.828+0000 7f7c2d63e640 10 monclient(hunting): authenticate will time out at 8388.140630s 2024-03-25T18:28:29.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.837+0000 7f7c26ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7c2806ab80 0x7f7c280724e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:53078/0 (socket says 172.21.15.176:53078) 2024-03-25T18:28:29.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.837+0000 7f7c26ffd640 1 -- 172.21.15.176:0/1269302596 learned_addr learned my addr 172.21.15.176:0/1269302596 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121995143 0 0) 0x7f7c280afce0 con 0x7f7c2806ab80 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:28:29.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.845+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c14003520 con 0x7f7c2806ab80 2024-03-25T18:28:29.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592621100 0 0) 0x7f7c14003520 con 0x7f7c2806ab80 2024-03-25T18:28:29.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 >> v1:172.21.15.176:6790/0 conn(0x7f7c2813f990 legacy=0x7f7c2814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 >> v1:172.21.15.176:6789/0 conn(0x7f7c2806af60 legacy=0x7f7c281458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 monclient: found mon.noname-c 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c280aff20 con 0x7f7c2806ab80 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c18001690 con 0x7f7c2806ab80 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:29.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c267fc640 10 monclient: dump: 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.856 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.857 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.857 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c2d63e640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:28:29.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.851+0000 7f7c2d63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:29.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c180017f0 con 0x7f7c2806ab80 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 1 -- 172.21.15.176:0/1269302596 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c18001c00 con 0x7f7c2806ab80 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c2d63e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c2d63e640 4 set_mon_vals no callback set 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:29.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.862+0000 7f7c267fc640 10 monclient: dump: 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.866 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 10 monclient: shutdown 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1269302596 >> v1:172.21.15.186:6789/0 conn(0x7f7c2806ab80 legacy=0x7f7c280724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1269302596 shutdown_connections 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1269302596 >> 172.21.15.176:0/1269302596 conn(0x7f7c2806c830 msgr2=0x7f7c280b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1269302596 shutdown_connections 2024-03-25T18:28:29.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.863+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1269302596 wait complete. 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.864+0000 7f7c2d63e640 10 monclient: build_initial_monmap 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.864+0000 7f7c2d63e640 10 monclient: monmap: 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.864+0000 7f7c2d63e640 1 librados: starting msgr at 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.864+0000 7f7c2d63e640 1 librados: starting objecter 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 Processor -- start 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 -- start start 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 librados: setting wanted keys 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 librados: calling monclient init 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient: init 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-a con 0x7f7c2806ab80 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-b con 0x7f7c2813f990 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient(hunting): picked mon.noname-c con 0x7f7c281d7ce0 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2813c020 con 0x7f7c2806ab80 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c281d8130 con 0x7f7c2813f990 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2814eff0 con 0x7f7c281d7ce0 2024-03-25T18:28:29.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:29.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.865+0000 7f7c2d63e640 10 monclient(hunting): authenticate will time out at 8388.177621s 2024-03-25T18:28:29.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.879+0000 7f7c277fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f7c281d7ce0 0x7f7c281e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:37490/0 (socket says 172.21.15.176:37490) 2024-03-25T18:28:29.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.879+0000 7f7c277fe640 1 -- 172.21.15.176:0/1507347875 learned_addr learned my addr 172.21.15.176:0/1507347875 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:29.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.884+0000 7f7c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c281d7ce0 2024-03-25T18:28:29.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.884+0000 7f7c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:29.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.885+0000 7f7c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c2813f990 2024-03-25T18:28:29.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.885+0000 7f7c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:29.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.895+0000 7f7c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c2806ab80 2024-03-25T18:28:29.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.895+0000 7f7c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920568320 0 0) 0x7f7c281d8130 con 0x7f7c2813f990 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:28:29.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.896+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c08004eb0 con 0x7f7c2813f990 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330277974 0 0) 0x7f7c2814eff0 con 0x7f7c281d7ce0 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:28:29.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.900+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c281d8130 con 0x7f7c281d7ce0 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432651540 0 0) 0x7f7c2813c020 con 0x7f7c2806ab80 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:28:29.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.909+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c2814eff0 con 0x7f7c2806ab80 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.927+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042132549 0 0) 0x7f7c2814eff0 con 0x7f7c2806ab80 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c2813c020 con 0x7f7c2806ab80 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c18001b00 con 0x7f7c2806ab80 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.929 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.928+0000 7f7c1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491263040 0 0) 0x7f7c08004eb0 con 0x7f7c2813f990 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c2814eff0 con 0x7f7c2813f990 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c10001930 con 0x7f7c2813f990 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.929+0000 7f7c1f7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.931 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173995062 0 0) 0x7f7c281d8130 con 0x7f7c281d7ce0 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c08004eb0 con 0x7f7c281d7ce0 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c20001c10 con 0x7f7c281d7ce0 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.930+0000 7f7c1f7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.932 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.933 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.940+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3975003249 0 0) 0x7f7c2813c020 con 0x7f7c2806ab80 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.940+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 >> v1:172.21.15.176:6790/0 conn(0x7f7c281d7ce0 legacy=0x7f7c281e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 >> v1:172.21.15.186:6789/0 conn(0x7f7c2813f990 legacy=0x7f7c281d75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 1 monclient: found mon.a 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:29.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2813f700 con 0x7f7c2806ab80 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 5 monclient: authenticate success, global_id 4398 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 10 monclient: _renew_subs 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2813ae70 con 0x7f7c2806ab80 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 10 monclient: _renew_subs 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281e37a0 con 0x7f7c2806ab80 2024-03-25T18:28:29.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.941+0000 7f7c2d63e640 1 librados: init done 2024-03-25T18:28:29.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.942+0000 7f7c2d63e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:29.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.942+0000 7f7c2d63e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:29.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.942+0000 7f7c2d63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:29.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.942+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c280627b0 con 0x7f7c2806ab80 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c18001c60 con 0x7f7c2806ab80 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7c18004c30 con 0x7f7c2806ab80 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c1f7fe640 10 monclient: dump: 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:29.962 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:29.963 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.961+0000 7f7c25ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:29.964 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.962+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f7c1801ac00 con 0x7f7c2806ab80 2024-03-25T18:28:29.964 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.962+0000 7f7c1ffff640 10 monclient: get_auth_request con 0x7f7c0806e0f0 auth_method 0 2024-03-25T18:28:29.964 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.962+0000 7f7c1f7fe640 10 client.4398.objecter ms_handle_connect 0x7f7c0806e0f0 2024-03-25T18:28:29.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.965+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7c18054820 con 0x7f7c2806ab80 2024-03-25T18:28:29.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.965+0000 7f7c1f7fe640 10 client.4398.objecter ms_dispatch 0x7f7c28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:29.967 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.965+0000 7f7c1f7fe640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:29.967 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.965+0000 7f7c1f7fe640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:29.967 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:29.965+0000 7f7c1f7fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T18:28:30.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.251+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c18050330 con 0x7f7c2806ab80 2024-03-25T18:28:30.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.251+0000 7f7c1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:30.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.251+0000 7f7c1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:30.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.480+0000 7f7c2d63e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:30.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.480+0000 7f7c2d63e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:30.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.480+0000 7f7c2d63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:30.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.480+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c28069f40 con 0x7f7c2806ab80 2024-03-25T18:28:30.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.524+0000 7f7c1f7fe640 1 -- 172.21.15.176:0/1507347875 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7c18052260 con 0x7f7c2806ab80 2024-03-25T18:28:30.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.524+0000 7f7c1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:30.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.524+0000 7f7c1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:30.526 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:30.526 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.526+0000 7f7c2d63e640 10 librados: watch_flush enter 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.526+0000 7f7c2d63e640 10 librados: watch_flush exit 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 10 client.4398.objecter successfully canceled tick 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 >> v1:172.21.15.176:6800/2598213983 conn(0x7f7c0806e0f0 legacy=0x7f7c080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 10 monclient: shutdown 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 >> v1:172.21.15.176:6789/0 conn(0x7f7c2806ab80 legacy=0x7f7c2813b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 shutdown_connections 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 >> 172.21.15.176:0/1507347875 conn(0x7f7c2806c830 msgr2=0x7f7c280b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 shutdown_connections 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 -- 172.21.15.176:0/1507347875 wait complete. 2024-03-25T18:28:30.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:30.527+0000 7f7c2d63e640 1 librados: shutdown 2024-03-25T18:28:30.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:37.351 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.466+0000 7fbf8817b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.467+0000 7fbf8817b640 10 monclient: build_initial_monmap 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.467+0000 7fbf8817b640 10 monclient: monmap: 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:37.468876+0000 2024-03-25T18:28:37.469 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:37.468876+0000 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.467+0000 7fbf8817b640 1 Processor -- start 2024-03-25T18:28:37.471 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.467+0000 7fbf8817b640 1 -- start start 2024-03-25T18:28:37.471 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.467+0000 7fbf8817b640 10 monclient: init 2024-03-25T18:28:37.471 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-b con 0x7fbf800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-c con 0x7fbf8013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-a con 0x7fbf801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf80142520 con 0x7fbf801430c0 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf800b28a0 con 0x7fbf8013f710 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.468+0000 7fbf8817b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf800b2a30 con 0x7fbf800a98f0 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.469+0000 7fbf8817b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:37.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.469+0000 7fbf8817b640 10 monclient(hunting): authenticate will time out at 8395.781847s 2024-03-25T18:28:37.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.471+0000 7fbf85f17640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fbf800a98f0 0x7fbf800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:43114/0 (socket says 172.21.15.176:43114) 2024-03-25T18:28:37.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.471+0000 7fbf85f17640 1 -- 172.21.15.176:0/983980426 learned_addr learned my addr 172.21.15.176:0/983980426 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:37.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965622419 0 0) 0x7fbf800b2a30 con 0x7fbf800a98f0 2024-03-25T18:28:37.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.494 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:28:37.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.492+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c003520 con 0x7fbf800a98f0 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15234216 0 0) 0x7fbf80142520 con 0x7fbf801430c0 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:28:37.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.496+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf800b2a30 con 0x7fbf801430c0 2024-03-25T18:28:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822134105 0 0) 0x7fbf800b28a0 con 0x7fbf8013f710 2024-03-25T18:28:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:28:37.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.497+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80142520 con 0x7fbf8013f710 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844304203 0 0) 0x7fbf6c003520 con 0x7fbf800a98f0 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 >> v1:172.21.15.186:6789/0 conn(0x7fbf8013f710 legacy=0x7fbf801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 >> v1:172.21.15.176:6789/0 conn(0x7fbf801430c0 legacy=0x7fbf8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: _finish_hunting 0 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 monclient: found mon.noname-b 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800b44c0 con 0x7fbf800a98f0 2024-03-25T18:28:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: _finish_auth 0 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: _check_auth_tickets 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbf7c001690 con 0x7fbf800a98f0 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.501+0000 7fbf84f15640 10 monclient: dump: 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:37.505 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:37.506 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.502+0000 7fbf8817b640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:28:37.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.502+0000 7fbf8817b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:37.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf7c0017f0 con 0x7fbf800a98f0 2024-03-25T18:28:37.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:37.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 1 -- 172.21.15.176:0/983980426 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbf7c001c00 con 0x7fbf800a98f0 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf84f15640 10 monclient: dump: 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf8817b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:37.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.536+0000 7fbf8817b640 4 set_mon_vals no callback set 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 10 monclient: shutdown 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 1 -- 172.21.15.176:0/983980426 >> v1:172.21.15.176:6790/0 conn(0x7fbf800a98f0 legacy=0x7fbf800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 1 -- 172.21.15.176:0/983980426 shutdown_connections 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 1 -- 172.21.15.176:0/983980426 >> 172.21.15.176:0/983980426 conn(0x7fbf800b7ea0 msgr2=0x7fbf800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 1 -- 172.21.15.176:0/983980426 shutdown_connections 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.537+0000 7fbf8817b640 1 -- 172.21.15.176:0/983980426 wait complete. 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 10 monclient: build_initial_monmap 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 10 monclient: monmap: 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:37.542 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 librados: starting msgr at 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 librados: starting objecter 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 Processor -- start 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 -- start start 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 librados: setting wanted keys 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 1 librados: calling monclient init 2024-03-25T18:28:37.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.538+0000 7fbf8817b640 10 monclient: init 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-c con 0x7fbf800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient(hunting): picked mon.noname-a con 0x7fbf801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf80138b20 con 0x7fbf801430c0 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf80137a80 con 0x7fbf8013f710 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf80139880 con 0x7fbf800a98f0 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:37.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.539+0000 7fbf8817b640 10 monclient(hunting): authenticate will time out at 8395.852384s 2024-03-25T18:28:37.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.553+0000 7fbf85f17640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fbf800a98f0 0x7fbf80137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:43156/0 (socket says 172.21.15.176:43156) 2024-03-25T18:28:37.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.553+0000 7fbf85f17640 1 -- 172.21.15.176:0/2669024841 learned_addr learned my addr 172.21.15.176:0/2669024841 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:37.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.554+0000 7fbf767fc640 10 client.?.objecter ms_handle_connect 0x7fbf800a98f0 2024-03-25T18:28:37.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.554+0000 7fbf767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:37.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.557+0000 7fbf767fc640 10 client.?.objecter ms_handle_connect 0x7fbf801430c0 2024-03-25T18:28:37.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.557+0000 7fbf767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210639371 0 0) 0x7fbf80139880 con 0x7fbf800a98f0 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:28:37.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.569+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf64004eb0 con 0x7fbf800a98f0 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 client.?.objecter ms_handle_connect 0x7fbf8013f710 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094094128 0 0) 0x7fbf80138b20 con 0x7fbf801430c0 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:28:37.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.573+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80139880 con 0x7fbf801430c0 2024-03-25T18:28:37.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005164460 0 0) 0x7fbf80137a80 con 0x7fbf8013f710 2024-03-25T18:28:37.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:37.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:37.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:37.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:28:37.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.581+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80138b20 con 0x7fbf8013f710 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608129727 0 0) 0x7fbf80139880 con 0x7fbf801430c0 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf80137a80 con 0x7fbf801430c0 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbf78001b50 con 0x7fbf801430c0 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 10 monclient(hunting): dump: 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.592 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.590+0000 7fbf767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289052142 0 0) 0x7fbf80138b20 con 0x7fbf8013f710 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf80139880 con 0x7fbf8013f710 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbf70001830 con 0x7fbf8013f710 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.600+0000 7fbf767fc640 10 monclient(hunting): dump: 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:37.602 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:37.603 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3013384625 0 0) 0x7fbf80137a80 con 0x7fbf801430c0 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 >> v1:172.21.15.176:6790/0 conn(0x7fbf800a98f0 legacy=0x7fbf80137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 >> v1:172.21.15.186:6789/0 conn(0x7fbf8013f710 legacy=0x7fbf80139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 1 monclient: found mon.a 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:37.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80139a10 con 0x7fbf801430c0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 10 monclient: _renew_subs 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf801dab00 con 0x7fbf801430c0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 10 monclient: _renew_subs 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8013b4f0 con 0x7fbf801430c0 2024-03-25T18:28:37.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.606+0000 7fbf8817b640 1 librados: init done 2024-03-25T18:28:37.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.607+0000 7fbf8817b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:37.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.607+0000 7fbf8817b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:37.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.607+0000 7fbf8817b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:37.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.607+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf801dae00 con 0x7fbf801430c0 2024-03-25T18:28:37.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf78002060 con 0x7fbf801430c0 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbf78003b10 con 0x7fbf801430c0 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf767fc640 10 monclient: dump: 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:37.911 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.908+0000 7fbf77fff640 4 set_mon_vals no callback set 2024-03-25T18:28:37.912 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.909+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fbf78019ae0 con 0x7fbf801430c0 2024-03-25T18:28:37.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.910+0000 7fbf85f17640 10 monclient: get_auth_request con 0x7fbf6406de40 auth_method 0 2024-03-25T18:28:37.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.910+0000 7fbf767fc640 10 client.4404.objecter ms_handle_connect 0x7fbf6406de40 2024-03-25T18:28:37.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fbf780538f0 con 0x7fbf801430c0 2024-03-25T18:28:37.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 10 client.4404.objecter ms_dispatch 0x7fbf80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:37.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:37.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:37.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:28:37.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf7804f230 con 0x7fbf801430c0 2024-03-25T18:28:37.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:37.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:37.915+0000 7fbf767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.124+0000 7fbf8817b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.125+0000 7fbf8817b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.125+0000 7fbf8817b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:38.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.125+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf800b3e80 con 0x7fbf801430c0 2024-03-25T18:28:38.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.163+0000 7fbf767fc640 1 -- 172.21.15.176:0/2669024841 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fbf78051330 con 0x7fbf801430c0 2024-03-25T18:28:38.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.163+0000 7fbf767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:38.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.163+0000 7fbf767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:38.165 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:38.165 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:38.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 10 librados: watch_flush enter 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 10 librados: watch_flush exit 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 >> v1:172.21.15.176:6800/2598213983 conn(0x7fbf6406de40 legacy=0x7fbf64070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 10 monclient: shutdown 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 >> v1:172.21.15.176:6789/0 conn(0x7fbf801430c0 legacy=0x7fbf801e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 shutdown_connections 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 >> 172.21.15.176:0/2669024841 conn(0x7fbf800b7ea0 msgr2=0x7fbf80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 shutdown_connections 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 -- 172.21.15.176:0/2669024841 wait complete. 2024-03-25T18:28:38.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:38.165+0000 7fbf8817b640 1 librados: shutdown 2024-03-25T18:28:38.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:44.988 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.101+0000 7f1343def640 10 monclient: get_monmap_and_config 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.101+0000 7f1343def640 10 monclient: build_initial_monmap 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.101+0000 7f1343def640 10 monclient: monmap: 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:45.103636+0000 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:45.103636+0000 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:45.104 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.102+0000 7f1343def640 1 Processor -- start 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.102+0000 7f1343def640 1 -- start start 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.102+0000 7f1343def640 10 monclient: init 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient(hunting): picked mon.noname-a con 0x7f133c0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient(hunting): picked mon.noname-b con 0x7f133c13f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient(hunting): picked mon.noname-c con 0x7f133c1430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c142520 con 0x7f133c0a98f0 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b28a0 con 0x7f133c1430c0 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b2a30 con 0x7f133c13f710 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:45.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.103+0000 7f1343def640 10 monclient(hunting): authenticate will time out at 8403.416485s 2024-03-25T18:28:45.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.116+0000 7f134138a640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f133c13f710 0x7f133c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:49602/0 (socket says 172.21.15.176:49602) 2024-03-25T18:28:45.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.116+0000 7f134138a640 1 -- 172.21.15.176:0/2626890151 learned_addr learned my addr 172.21.15.176:0/2626890151 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:45.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435626085 0 0) 0x7f133c0b2a30 con 0x7f133c13f710 2024-03-25T18:28:45.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:28:45.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.127+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1328003520 con 0x7f133c13f710 2024-03-25T18:28:45.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726179743 0 0) 0x7f133c0b28a0 con 0x7f133c1430c0 2024-03-25T18:28:45.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c0b2a30 con 0x7f133c1430c0 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555754638 0 0) 0x7f133c142520 con 0x7f133c0a98f0 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:28:45.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.131+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c0b28a0 con 0x7f133c0a98f0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694882335 0 0) 0x7f133c0b2a30 con 0x7f133c1430c0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 >> v1:172.21.15.176:6790/0 conn(0x7f133c13f710 legacy=0x7f133c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 >> v1:172.21.15.176:6789/0 conn(0x7f133c0a98f0 legacy=0x7f133c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: _finish_hunting 0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 monclient: found mon.noname-c 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0b44c0 con 0x7f133c1430c0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: _finish_auth 0 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: _check_auth_tickets 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:45.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1343def640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f13340019f0 con 0x7f133c1430c0 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1340b89640 10 monclient: dump: 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.147+0000 7f1343def640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1334001b50 con 0x7f133c1430c0 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 1 -- 172.21.15.176:0/2626890151 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1334001f60 con 0x7f133c1430c0 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1343def640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1343def640 4 set_mon_vals no callback set 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1340b89640 10 monclient: dump: 2024-03-25T18:28:45.162 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:45.163 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1343def640 10 monclient: shutdown 2024-03-25T18:28:45.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1343def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:45.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.160+0000 7f1343def640 1 -- 172.21.15.176:0/2626890151 >> v1:172.21.15.186:6789/0 conn(0x7f133c1430c0 legacy=0x7f133c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 1 -- 172.21.15.176:0/2626890151 shutdown_connections 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 1 -- 172.21.15.176:0/2626890151 >> 172.21.15.176:0/2626890151 conn(0x7f133c0b7ea0 msgr2=0x7f133c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 1 -- 172.21.15.176:0/2626890151 shutdown_connections 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 1 -- 172.21.15.176:0/2626890151 wait complete. 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 10 monclient: build_initial_monmap 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.161+0000 7f1343def640 10 monclient: monmap: 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:45.166 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 librados: starting msgr at 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 librados: starting objecter 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 Processor -- start 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 -- start start 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 librados: setting wanted keys 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 librados: calling monclient init 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient: init 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient(hunting): picked mon.noname-b con 0x7f133c0a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient(hunting): picked mon.noname-c con 0x7f133c13f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 10 monclient(hunting): picked mon.noname-a con 0x7f133c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c1dac60 con 0x7f133c1430c0 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c1da8f0 con 0x7f133c0a98f0 2024-03-25T18:28:45.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.162+0000 7f1343def640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c138ec0 con 0x7f133c13f710 2024-03-25T18:28:45.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.163+0000 7f1343def640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:45.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.163+0000 7f1343def640 10 monclient(hunting): authenticate will time out at 8403.475899s 2024-03-25T18:28:45.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.175+0000 7f134238c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f133c1430c0 0x7f133c1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:46204/0 (socket says 172.21.15.176:46204) 2024-03-25T18:28:45.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.175+0000 7f134238c640 1 -- 172.21.15.176:0/823952009 learned_addr learned my addr 172.21.15.176:0/823952009 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:45.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.182+0000 7f13327fc640 10 client.?.objecter ms_handle_connect 0x7f133c1430c0 2024-03-25T18:28:45.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.182+0000 7f13327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.182+0000 7f13327fc640 10 client.?.objecter ms_handle_connect 0x7f133c0a98f0 2024-03-25T18:28:45.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.182+0000 7f13327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.185+0000 7f13327fc640 10 client.?.objecter ms_handle_connect 0x7f133c13f710 2024-03-25T18:28:45.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.185+0000 7f13327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376797865 0 0) 0x7f133c1da8f0 con 0x7f133c0a98f0 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:28:45.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.192+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320004eb0 con 0x7f133c0a98f0 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.199+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013925616 0 0) 0x7f133c1dac60 con 0x7f133c1430c0 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.199+0000 7f13327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.199+0000 7f13327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.199+0000 7f13327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.199+0000 7f13327fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:28:45.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c1da8f0 con 0x7f133c1430c0 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269029224 0 0) 0x7f1320004eb0 con 0x7f133c0a98f0 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133c1dac60 con 0x7f133c0a98f0 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f13380018f0 con 0x7f133c0a98f0 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 10 monclient(hunting): dump: 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:45.203 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.200+0000 7f13327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507875638 0 0) 0x7f133c138ec0 con 0x7f133c13f710 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:28:45.204 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.202+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320004eb0 con 0x7f133c13f710 2024-03-25T18:28:45.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2650310586 0 0) 0x7f133c1dac60 con 0x7f133c0a98f0 2024-03-25T18:28:45.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 >> v1:172.21.15.176:6790/0 conn(0x7f133c13f710 legacy=0x7f133c1382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 >> v1:172.21.15.176:6789/0 conn(0x7f133c1430c0 legacy=0x7f133c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 1 monclient: found mon.b 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c139050 con 0x7f133c0a98f0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f13327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 10 monclient: _renew_subs 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c137970 con 0x7f133c0a98f0 2024-03-25T18:28:45.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 10 monclient: _renew_subs 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.213+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c138b90 con 0x7f133c0a98f0 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.214+0000 7f1343def640 1 librados: init done 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.214+0000 7f1343def640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.214+0000 7f1343def640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.214+0000 7f1343def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.214+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c0627b0 con 0x7f133c0a98f0 2024-03-25T18:28:45.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1338003610 con 0x7f133c0a98f0 2024-03-25T18:28:45.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:45.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f13380039a0 con 0x7f133c0a98f0 2024-03-25T18:28:45.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f13327fc640 10 monclient: dump: 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:45.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.485+0000 7f1333fff640 4 set_mon_vals no callback set 2024-03-25T18:28:45.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.486+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f1338003bc0 con 0x7f133c0a98f0 2024-03-25T18:28:45.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.486+0000 7f134138a640 10 monclient: get_auth_request con 0x7f132006de00 auth_method 0 2024-03-25T18:28:45.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.486+0000 7f13327fc640 10 client.4420.objecter ms_handle_connect 0x7f132006de00 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f13380535a0 con 0x7f133c0a98f0 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 10 client.4420.objecter ms_dispatch 0x7f133c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f133804f0b0 con 0x7f133c0a98f0 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:45.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.493+0000 7f13327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:45.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.725+0000 7f1343def640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:45.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.725+0000 7f1343def640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:45.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.725+0000 7f1343def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:28:45.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.725+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f133c1dadf0 con 0x7f133c0a98f0 2024-03-25T18:28:45.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.802+0000 7f13327fc640 1 -- 172.21.15.176:0/823952009 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f1338050fe0 con 0x7f133c0a98f0 2024-03-25T18:28:45.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.802+0000 7f13327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:45.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.802+0000 7f13327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:45.805 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:45.805 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 10 librados: watch_flush enter 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 10 librados: watch_flush exit 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 >> v1:172.21.15.176:6800/2598213983 conn(0x7f132006de00 legacy=0x7f13200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 10 monclient: shutdown 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 >> v1:172.21.15.186:6789/0 conn(0x7f133c0a98f0 legacy=0x7f133c1db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 shutdown_connections 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 >> 172.21.15.176:0/823952009 conn(0x7f133c0b7ea0 msgr2=0x7f133c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:45.808 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 shutdown_connections 2024-03-25T18:28:45.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.805+0000 7f1343def640 1 -- 172.21.15.176:0/823952009 wait complete. 2024-03-25T18:28:45.809 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:45.806+0000 7f1343def640 1 librados: shutdown 2024-03-25T18:28:45.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:48.156 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:49.911 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.170 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:49.926 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.210 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:04:49.966 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.223 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:49.957 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.228 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:49.963 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.234 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:49.990 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.240 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:49.997 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.250 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:49.985 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.266 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:04:50.022 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:52.627 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:52.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.739+0000 7ff34f2c9640 10 monclient: get_monmap_and_config 2024-03-25T18:28:52.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.739+0000 7ff34f2c9640 10 monclient: build_initial_monmap 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.739+0000 7ff34f2c9640 10 monclient: monmap: 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:28:52.740437+0000 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:28:52.740437+0000 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:28:52.741 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.740+0000 7ff34f2c9640 1 Processor -- start 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.740+0000 7ff34f2c9640 1 -- start start 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.740+0000 7ff34f2c9640 10 monclient: init 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff3480a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff34813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff3481430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff348142520 con 0x7ff3480a98f0 2024-03-25T18:28:52.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480b28a0 con 0x7ff3481430c0 2024-03-25T18:28:52.744 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480b2a30 con 0x7ff34813f710 2024-03-25T18:28:52.744 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:52.744 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.741+0000 7ff34f2c9640 10 monclient(hunting): authenticate will time out at 8411.053653s 2024-03-25T18:28:52.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.747+0000 7ff34d866640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff3481430c0 0x7ff34814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:52160/0 (socket says 172.21.15.176:52160) 2024-03-25T18:28:52.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.747+0000 7ff34d866640 1 -- 172.21.15.176:0/1327881399 learned_addr learned my addr 172.21.15.176:0/1327881399 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804818273 0 0) 0x7ff3480b28a0 con 0x7ff3481430c0 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:28:52.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.764+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff334003520 con 0x7ff3481430c0 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568013206 0 0) 0x7ff3480b2a30 con 0x7ff34813f710 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:28:52.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.766+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3480b28a0 con 0x7ff34813f710 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.774+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648960492 0 0) 0x7ff334003520 con 0x7ff3481430c0 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.774+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 >> v1:172.21.15.176:6790/0 conn(0x7ff34813f710 legacy=0x7ff3481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 >> v1:172.21.15.176:6789/0 conn(0x7ff3480a98f0 legacy=0x7ff3480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 1 monclient: found mon.noname-c 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:52.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480b44c0 con 0x7ff3481430c0 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: _finish_auth 0 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff340001a10 con 0x7ff3481430c0 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff34f2c9640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff34f2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff33ffff640 10 monclient: dump: 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.777 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.775+0000 7ff34f2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff340001b70 con 0x7ff3481430c0 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 1 -- 172.21.15.176:0/1327881399 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff340001f80 con 0x7ff3481430c0 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff33ffff640 10 monclient: dump: 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:52.796 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff34f2c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:52.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.795+0000 7ff34f2c9640 4 set_mon_vals no callback set 2024-03-25T18:28:52.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 10 monclient: shutdown 2024-03-25T18:28:52.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:52.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 1 -- 172.21.15.176:0/1327881399 >> v1:172.21.15.186:6789/0 conn(0x7ff3481430c0 legacy=0x7ff34814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 1 -- 172.21.15.176:0/1327881399 shutdown_connections 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 1 -- 172.21.15.176:0/1327881399 >> 172.21.15.176:0/1327881399 conn(0x7ff3480b7ea0 msgr2=0x7ff3480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 1 -- 172.21.15.176:0/1327881399 shutdown_connections 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.796+0000 7ff34f2c9640 1 -- 172.21.15.176:0/1327881399 wait complete. 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 10 monclient: build_initial_monmap 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 10 monclient: monmap: 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:28:52.800 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 librados: starting msgr at 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 librados: starting objecter 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 Processor -- start 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 -- start start 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 librados: setting wanted keys 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 1 librados: calling monclient init 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.797+0000 7ff34f2c9640 10 monclient: init 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff3480a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff34813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff3481430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:28:52.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3481369d0 con 0x7ff3481430c0 2024-03-25T18:28:52.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff348133a60 con 0x7ff34813f710 2024-03-25T18:28:52.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480ab9e0 con 0x7ff3480a98f0 2024-03-25T18:28:52.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:52.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.798+0000 7ff34f2c9640 10 monclient(hunting): authenticate will time out at 8411.110112s 2024-03-25T18:28:52.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.819+0000 7ff34d866640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff3481430c0 0x7ff348133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:50878/0 (socket says 172.21.15.176:50878) 2024-03-25T18:28:52.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.819+0000 7ff34d866640 1 -- 172.21.15.176:0/3901925885 learned_addr learned my addr 172.21.15.176:0/3901925885 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:28:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.820+0000 7ff33dffb640 10 client.?.objecter ms_handle_connect 0x7ff3481430c0 2024-03-25T18:28:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.820+0000 7ff33dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.821+0000 7ff33dffb640 10 client.?.objecter ms_handle_connect 0x7ff3480a98f0 2024-03-25T18:28:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.821+0000 7ff33dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:52.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.826+0000 7ff33dffb640 10 client.?.objecter ms_handle_connect 0x7ff34813f710 2024-03-25T18:28:52.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.826+0000 7ff33dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323271810 0 0) 0x7ff3480ab9e0 con 0x7ff3480a98f0 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff32c004eb0 con 0x7ff3480a98f0 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427799770 0 0) 0x7ff3481369d0 con 0x7ff3481430c0 2024-03-25T18:28:52.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:52.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:52.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:52.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:28:52.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.829+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3480ab9e0 con 0x7ff3481430c0 2024-03-25T18:28:52.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178208120 0 0) 0x7ff348133a60 con 0x7ff34813f710 2024-03-25T18:28:52.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:52.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:52.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:52.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:28:52.834 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.833+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3481369d0 con 0x7ff34813f710 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780855936 0 0) 0x7ff3480ab9e0 con 0x7ff3481430c0 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff348133a60 con 0x7ff3481430c0 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff340001e80 con 0x7ff3481430c0 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 10 monclient(hunting): dump: 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.843+0000 7ff33dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:52.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062572924 0 0) 0x7ff32c004eb0 con 0x7ff3480a98f0 2024-03-25T18:28:52.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3480ab9e0 con 0x7ff3480a98f0 2024-03-25T18:28:52.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff344001930 con 0x7ff3480a98f0 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.852+0000 7ff33dffb640 10 monclient(hunting): dump: 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.854 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:52.855 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.855 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:52.855 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:52.855 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:52.855 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032785844 0 0) 0x7ff3481369d0 con 0x7ff34813f710 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff32c004eb0 con 0x7ff34813f710 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff338001930 con 0x7ff34813f710 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.859+0000 7ff33dffb640 10 monclient(hunting): dump: 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:52.860 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:52.861 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.861+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2353032673 0 0) 0x7ff348133a60 con 0x7ff3481430c0 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 >> v1:172.21.15.176:6790/0 conn(0x7ff3480a98f0 legacy=0x7ff348132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 >> v1:172.21.15.186:6789/0 conn(0x7ff34813f710 legacy=0x7ff3481362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 1 monclient: found mon.a 2024-03-25T18:28:52.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480abcc0 con 0x7ff3481430c0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff33dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 10 monclient: _renew_subs 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3480abfa0 con 0x7ff3481430c0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 10 monclient: _renew_subs 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:52.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff348137e40 con 0x7ff3481430c0 2024-03-25T18:28:52.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.862+0000 7ff34f2c9640 1 librados: init done 2024-03-25T18:28:52.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.863+0000 7ff34f2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:52.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.863+0000 7ff34f2c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:52.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.863+0000 7ff34f2c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:52.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:52.863+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff348139060 con 0x7ff3481430c0 2024-03-25T18:28:53.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff340001570 con 0x7ff3481430c0 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff340004e10 con 0x7ff3481430c0 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33dffb640 10 monclient: dump: 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:28:53.174 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.172+0000 7ff33f7fe640 4 set_mon_vals no callback set 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.173+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7ff34001aea0 con 0x7ff3481430c0 2024-03-25T18:28:53.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.173+0000 7ff34d065640 10 monclient: get_auth_request con 0x7ff32c06e0b0 auth_method 0 2024-03-25T18:28:53.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.174+0000 7ff33dffb640 10 client.4416.objecter ms_handle_connect 0x7ff32c06e0b0 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7ff340054ae0 con 0x7ff3481430c0 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 10 client.4416.objecter ms_dispatch 0x7ff348000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff340050420 con 0x7ff3481430c0 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:53.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.179+0000 7ff33dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:53.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.437+0000 7ff34f2c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:53.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.437+0000 7ff34f2c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:53.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.437+0000 7ff34f2c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:28:53.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.437+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3480627b0 con 0x7ff3481430c0 2024-03-25T18:28:53.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.462+0000 7ff33dffb640 1 -- 172.21.15.176:0/3901925885 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7ff340052520 con 0x7ff3481430c0 2024-03-25T18:28:53.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.462+0000 7ff33dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:53.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.462+0000 7ff33dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:53.463 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:28:53.463 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:53.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 10 librados: watch_flush enter 2024-03-25T18:28:53.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 10 librados: watch_flush exit 2024-03-25T18:28:53.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 10 client.4416.objecter successfully canceled tick 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 >> v1:172.21.15.176:6800/2598213983 conn(0x7ff32c06e0b0 legacy=0x7ff32c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 10 monclient: shutdown 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 >> v1:172.21.15.176:6789/0 conn(0x7ff3481430c0 legacy=0x7ff348133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 shutdown_connections 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 >> 172.21.15.176:0/3901925885 conn(0x7ff3480b7ea0 msgr2=0x7ff348074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 shutdown_connections 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 -- 172.21.15.176:0/3901925885 wait complete. 2024-03-25T18:28:53.466 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:28:53.464+0000 7ff34f2c9640 1 librados: shutdown 2024-03-25T18:28:53.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:00.282 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.396+0000 7f25e1f89640 10 monclient: get_monmap_and_config 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.396+0000 7f25e1f89640 10 monclient: build_initial_monmap 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.396+0000 7f25e1f89640 10 monclient: monmap: 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:00.397021+0000 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:00.397021+0000 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:00.397 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:00.398 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:00.398 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:00.398 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:00.398 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.396+0000 7f25e1f89640 1 Processor -- start 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 1 -- start start 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient: init 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:00.399 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-a con 0x7f25dc0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-b con 0x7f25dc13f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-c con 0x7f25dc1430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc142520 con 0x7f25dc0a98f0 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0b28a0 con 0x7f25dc1430c0 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0b2a30 con 0x7f25dc13f710 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:00.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.397+0000 7f25e1f89640 10 monclient(hunting): authenticate will time out at 8418.709743s 2024-03-25T18:29:00.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.406+0000 7f25daffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f25dc13f710 0x7f25dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:42094/0 (socket says 172.21.15.176:42094) 2024-03-25T18:29:00.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.406+0000 7f25daffd640 1 -- 172.21.15.176:0/3666462436 learned_addr learned my addr 172.21.15.176:0/3666462436 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612152023 0 0) 0x7f25dc0b2a30 con 0x7f25dc13f710 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:29:00.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.421+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c4003520 con 0x7f25dc13f710 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814371256 0 0) 0x7f25dc0b28a0 con 0x7f25dc1430c0 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:29:00.427 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.426+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc0b2a30 con 0x7f25dc1430c0 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947161673 0 0) 0x7f25dc142520 con 0x7f25dc0a98f0 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc0b28a0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459896001 0 0) 0x7f25c4003520 con 0x7f25dc13f710 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 >> v1:172.21.15.186:6789/0 conn(0x7f25dc1430c0 legacy=0x7f25dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:00.431 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 >> v1:172.21.15.176:6789/0 conn(0x7f25dc0a98f0 legacy=0x7f25dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 monclient: found mon.noname-b 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc0b44c0 con 0x7f25dc13f710 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25d0001690 con 0x7f25dc13f710 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25e1f89640 5 monclient: authenticate success, global_id 4439 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25e1f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:00.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25da7fc640 10 monclient: dump: 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.433 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.429+0000 7f25e1f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:00.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d0001870 con 0x7f25dc13f710 2024-03-25T18:29:00.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:00.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 1 -- 172.21.15.176:0/3666462436 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25d0001c80 con 0x7f25dc13f710 2024-03-25T18:29:00.455 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25e1f89640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25e1f89640 4 set_mon_vals no callback set 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.454+0000 7f25da7fc640 10 monclient: dump: 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:00.456 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:00.457 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:00.457 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:00.457 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 10 monclient: shutdown 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 1 -- 172.21.15.176:0/3666462436 >> v1:172.21.15.176:6790/0 conn(0x7f25dc13f710 legacy=0x7f25dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 1 -- 172.21.15.176:0/3666462436 shutdown_connections 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 1 -- 172.21.15.176:0/3666462436 >> 172.21.15.176:0/3666462436 conn(0x7f25dc0b7ea0 msgr2=0x7f25dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 1 -- 172.21.15.176:0/3666462436 shutdown_connections 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 1 -- 172.21.15.176:0/3666462436 wait complete. 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 10 monclient: build_initial_monmap 2024-03-25T18:29:00.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.455+0000 7f25e1f89640 10 monclient: monmap: 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:00.461 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 librados: starting msgr at 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 librados: starting objecter 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 Processor -- start 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 -- start start 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 librados: setting wanted keys 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 1 librados: calling monclient init 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.456+0000 7f25e1f89640 10 monclient: init 2024-03-25T18:29:00.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-b con 0x7f25dc0a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-a con 0x7f25dc13f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient(hunting): picked mon.noname-c con 0x7f25dc1430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0ab8b0 con 0x7f25dc13f710 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0abb90 con 0x7f25dc0a98f0 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc137870 con 0x7f25dc1430c0 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:00.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.457+0000 7f25e1f89640 10 monclient(hunting): authenticate will time out at 8418.769387s 2024-03-25T18:29:00.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.461+0000 7f25db7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f25dc0a98f0 0x7f25dc1350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:37230/0 (socket says 172.21.15.176:37230) 2024-03-25T18:29:00.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.461+0000 7f25db7fe640 1 -- 172.21.15.176:0/2057520939 learned_addr learned my addr 172.21.15.176:0/2057520939 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:00.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.462+0000 7f25c3fff640 10 client.?.objecter ms_handle_connect 0x7f25dc0a98f0 2024-03-25T18:29:00.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.462+0000 7f25c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:00.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744004686 0 0) 0x7f25dc0abb90 con 0x7f25dc0a98f0 2024-03-25T18:29:00.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:00.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:00.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:00.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:29:00.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.468+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8004eb0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.469+0000 7f25c3fff640 10 client.?.objecter ms_handle_connect 0x7f25dc1430c0 2024-03-25T18:29:00.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.469+0000 7f25c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.479+0000 7f25c3fff640 10 client.?.objecter ms_handle_connect 0x7f25dc13f710 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.479+0000 7f25c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500490895 0 0) 0x7f25c8004eb0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc0abb90 con 0x7f25dc0a98f0 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25cc001a10 con 0x7f25dc0a98f0 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 10 monclient(hunting): dump: 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.481 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.482 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.480+0000 7f25c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:00.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2571266967 0 0) 0x7f25dc0abb90 con 0x7f25dc0a98f0 2024-03-25T18:29:00.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 >> v1:172.21.15.176:6790/0 conn(0x7f25dc1430c0 legacy=0x7f25dc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:00.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 >> v1:172.21.15.176:6789/0 conn(0x7f25dc13f710 legacy=0x7f25dc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:00.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 1 monclient: found mon.b 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc1389e0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 10 monclient: _renew_subs 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:00.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc136d90 con 0x7f25dc0a98f0 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 10 monclient: _renew_subs 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc1384c0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.486+0000 7f25e1f89640 1 librados: init done 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.487+0000 7f25e1f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.487+0000 7f25e1f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.487+0000 7f25e1f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:00.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.487+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc0627b0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25cc001cb0 con 0x7f25dc0a98f0 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25cc003a60 con 0x7f25dc0a98f0 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25c3fff640 10 monclient: dump: 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.766 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.764+0000 7f25d9ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:00.767 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.765+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f25cc003c80 con 0x7f25dc0a98f0 2024-03-25T18:29:00.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.766+0000 7f25daffd640 10 monclient: get_auth_request con 0x7f25c806dd90 auth_method 0 2024-03-25T18:29:00.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.766+0000 7f25c3fff640 10 client.4432.objecter ms_handle_connect 0x7f25c806dd90 2024-03-25T18:29:00.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f25cc054840 con 0x7f25dc0a98f0 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 10 client.4432.objecter ms_dispatch 0x7f25dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25cc050350 con 0x7f25dc0a98f0 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:00.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:00.771+0000 7f25c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:01.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.004+0000 7f25e1f89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.004+0000 7f25e1f89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.004+0000 7f25e1f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:01.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.004+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25dc138b70 con 0x7f25dc0a98f0 2024-03-25T18:29:01.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.039+0000 7f25c3fff640 1 -- 172.21.15.176:0/2057520939 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f25cc052280 con 0x7f25dc0a98f0 2024-03-25T18:29:01.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.039+0000 7f25c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:01.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.039+0000 7f25c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:01.040 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:01.040 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:01.042 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 10 librados: watch_flush enter 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 10 librados: watch_flush exit 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 >> v1:172.21.15.176:6800/2598213983 conn(0x7f25c806dd90 legacy=0x7f25c8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 10 monclient: shutdown 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 >> v1:172.21.15.186:6789/0 conn(0x7f25dc0a98f0 legacy=0x7f25dc1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 shutdown_connections 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.041+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 >> 172.21.15.176:0/2057520939 conn(0x7f25dc0b7ea0 msgr2=0x7f25dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.042+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 shutdown_connections 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.042+0000 7f25e1f89640 1 -- 172.21.15.176:0/2057520939 wait complete. 2024-03-25T18:29:01.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:01.042+0000 7f25e1f89640 1 librados: shutdown 2024-03-25T18:29:01.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:07.863 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:07.977 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.976+0000 7f8724651640 10 monclient: get_monmap_and_config 2024-03-25T18:29:07.977 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.976+0000 7f8724651640 10 monclient: build_initial_monmap 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.976+0000 7f8724651640 10 monclient: monmap: 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:07.977372+0000 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:07.977372+0000 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:07.978 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.977+0000 7f8724651640 1 Processor -- start 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.977+0000 7f8724651640 1 -- start start 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.977+0000 7f8724651640 10 monclient: init 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient(hunting): picked mon.noname-c con 0x7f871c0a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient(hunting): picked mon.noname-a con 0x7f871c13f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:07.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient(hunting): picked mon.noname-b con 0x7f871c1430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:07.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c142520 con 0x7f871c13f710 2024-03-25T18:29:07.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0b28a0 con 0x7f871c0a98f0 2024-03-25T18:29:07.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0b2a30 con 0x7f871c1430c0 2024-03-25T18:29:07.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:07.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.978+0000 7f8724651640 10 monclient(hunting): authenticate will time out at 8426.290594s 2024-03-25T18:29:07.991 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.990+0000 7f8722bee640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f871c1430c0 0x7f871c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:51194/0 (socket says 172.21.15.176:51194) 2024-03-25T18:29:07.992 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:07.991+0000 7f8722bee640 1 -- 172.21.15.176:0/2672218443 learned_addr learned my addr 172.21.15.176:0/2672218443 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903454598 0 0) 0x7f871c0b2a30 con 0x7f871c1430c0 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:29:08.005 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.004+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8708003520 con 0x7f871c1430c0 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245996052 0 0) 0x7f871c0b28a0 con 0x7f871c0a98f0 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:29:08.006 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.005+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c0b2a30 con 0x7f871c0a98f0 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.012+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910608565 0 0) 0x7f871c0b2a30 con 0x7f871c0a98f0 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.012+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 >> v1:172.21.15.176:6790/0 conn(0x7f871c1430c0 legacy=0x7f871c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 >> v1:172.21.15.176:6789/0 conn(0x7f871c13f710 legacy=0x7f871c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 1 monclient: found mon.noname-c 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:08.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0b44c0 con 0x7f871c0a98f0 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: _finish_auth 0 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8718001670 con 0x7f871c0a98f0 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f87213eb640 10 monclient: dump: 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:08.015 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:08.016 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:08.016 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:08.016 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:08.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f8724651640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:29:08.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.013+0000 7f8724651640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:08.020 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.019+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87180017d0 con 0x7f871c0a98f0 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.019+0000 7f87213eb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f87213eb640 1 -- 172.21.15.176:0/2672218443 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8718001ba0 con 0x7f871c0a98f0 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f8724651640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f8724651640 4 set_mon_vals no callback set 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f87213eb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f87213eb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f87213eb640 10 monclient: dump: 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.021 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:08.022 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:08.022 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:08.022 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:08.022 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f8724651640 10 monclient: shutdown 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f8724651640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.020+0000 7f8724651640 1 -- 172.21.15.176:0/2672218443 >> v1:172.21.15.186:6789/0 conn(0x7f871c0a98f0 legacy=0x7f871c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 1 -- 172.21.15.176:0/2672218443 shutdown_connections 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 1 -- 172.21.15.176:0/2672218443 >> 172.21.15.176:0/2672218443 conn(0x7f871c0b7ea0 msgr2=0x7f871c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 1 -- 172.21.15.176:0/2672218443 shutdown_connections 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 1 -- 172.21.15.176:0/2672218443 wait complete. 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 10 monclient: build_initial_monmap 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.021+0000 7f8724651640 10 monclient: monmap: 2024-03-25T18:29:08.024 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 librados: starting msgr at 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 librados: starting objecter 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 Processor -- start 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 -- start start 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 librados: setting wanted keys 2024-03-25T18:29:08.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 librados: calling monclient init 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient: init 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient(hunting): picked mon.noname-c con 0x7f871c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient(hunting): picked mon.noname-b con 0x7f871c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient(hunting): picked mon.noname-a con 0x7f871c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c1390e0 con 0x7f871c1430c0 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c1349e0 con 0x7f871c13f710 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c139350 con 0x7f871c0a98f0 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:08.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.022+0000 7f8724651640 10 monclient(hunting): authenticate will time out at 8426.334805s 2024-03-25T18:29:08.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.034+0000 7f8721bec640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f871c13f710 0x7f871c133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:52116/0 (socket says 172.21.15.176:52116) 2024-03-25T18:29:08.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.034+0000 7f8721bec640 1 -- 172.21.15.176:0/1189024025 learned_addr learned my addr 172.21.15.176:0/1189024025 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:08.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.038+0000 7f8712ffd640 10 client.?.objecter ms_handle_connect 0x7f871c13f710 2024-03-25T18:29:08.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.038+0000 7f8712ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.039+0000 7f8712ffd640 10 client.?.objecter ms_handle_connect 0x7f871c0a98f0 2024-03-25T18:29:08.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.039+0000 7f8712ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.043+0000 7f8712ffd640 10 client.?.objecter ms_handle_connect 0x7f871c1430c0 2024-03-25T18:29:08.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.043+0000 7f8712ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240042230 0 0) 0x7f871c139350 con 0x7f871c0a98f0 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:29:08.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.047+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8700004eb0 con 0x7f871c0a98f0 2024-03-25T18:29:08.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099196476 0 0) 0x7f871c1349e0 con 0x7f871c13f710 2024-03-25T18:29:08.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.056 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:29:08.057 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.055+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c139350 con 0x7f871c13f710 2024-03-25T18:29:08.063 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753795755 0 0) 0x7f871c1390e0 con 0x7f871c1430c0 2024-03-25T18:29:08.063 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:08.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:08.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:08.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:29:08.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.062+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c1349e0 con 0x7f871c1430c0 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.072+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897373395 0 0) 0x7f8700004eb0 con 0x7f871c0a98f0 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c1390e0 con 0x7f871c0a98f0 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8718001ae0 con 0x7f871c0a98f0 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.074 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:08.075 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:08.075 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:08.075 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:08.075 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:08.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.073+0000 7f8712ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997578327 0 0) 0x7f871c1390e0 con 0x7f871c0a98f0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 >> v1:172.21.15.186:6789/0 conn(0x7f871c13f710 legacy=0x7f871c133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 >> v1:172.21.15.176:6789/0 conn(0x7f871c1430c0 legacy=0x7f871c1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 1 monclient: found mon.c 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c139630 con 0x7f871c0a98f0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:08.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.086+0000 7f8712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _renew_subs 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c139910 con 0x7f871c0a98f0 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _renew_subs 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c137800 con 0x7f871c0a98f0 2024-03-25T18:29:08.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 1 librados: init done 2024-03-25T18:29:08.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:08.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:08.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:08.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.087+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c138410 con 0x7f871c0a98f0 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87180038a0 con 0x7f871c0a98f0 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8718003bb0 con 0x7f871c0a98f0 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:08.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 10 monclient: dump: 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8720bea640 4 set_mon_vals no callback set 2024-03-25T18:29:08.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.381+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f8718019b80 con 0x7f871c0a98f0 2024-03-25T18:29:08.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.382+0000 7f8721bec640 10 monclient: get_auth_request con 0x7f870006ddf0 auth_method 0 2024-03-25T18:29:08.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.382+0000 7f8712ffd640 10 client.4448.objecter ms_handle_connect 0x7f870006ddf0 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f8718053730 con 0x7f871c0a98f0 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 10 client.4448.objecter ms_dispatch 0x7f871c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.387+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f871804f240 con 0x7f871c0a98f0 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.388+0000 7f8712ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:08.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.388+0000 7f8712ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:08.613 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.612+0000 7f8724651640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:08.613 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.612+0000 7f8724651640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:08.613 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.612+0000 7f8724651640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:08.613 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.612+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f871c138910 con 0x7f871c0a98f0 2024-03-25T18:29:08.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.645+0000 7f8712ffd640 1 -- 172.21.15.176:0/1189024025 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f8718051170 con 0x7f871c0a98f0 2024-03-25T18:29:08.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.645+0000 7f8712ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:08.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.645+0000 7f8712ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:08.647 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:08.647 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:08.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.647+0000 7f8724651640 10 librados: watch_flush enter 2024-03-25T18:29:08.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.647+0000 7f8724651640 10 librados: watch_flush exit 2024-03-25T18:29:08.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.647+0000 7f8724651640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-25T18:29:08.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.647+0000 7f8724651640 10 client.4448.objecter successfully canceled tick 2024-03-25T18:29:08.649 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 >> v1:172.21.15.176:6800/2598213983 conn(0x7f870006ddf0 legacy=0x7f87000702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 10 monclient: shutdown 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 >> v1:172.21.15.176:6790/0 conn(0x7f871c0a98f0 legacy=0x7f871c1d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 shutdown_connections 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 >> 172.21.15.176:0/1189024025 conn(0x7f871c0b7ea0 msgr2=0x7f871c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 shutdown_connections 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 -- 172.21.15.176:0/1189024025 wait complete. 2024-03-25T18:29:08.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:08.648+0000 7f8724651640 1 librados: shutdown 2024-03-25T18:29:08.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:13.326 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:15.082 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.332 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:15.089 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.359 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:15.115 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.383 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:15.139 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.389 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:15.123 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.393 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:15.127 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.410 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:15.166 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.435 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:15.169 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.447 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:15.203 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:15.468 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:15.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.581+0000 7f9d75c96640 10 monclient: get_monmap_and_config 2024-03-25T18:29:15.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.581+0000 7f9d75c96640 10 monclient: build_initial_monmap 2024-03-25T18:29:15.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.581+0000 7f9d75c96640 10 monclient: monmap: 2024-03-25T18:29:15.582 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:15.582238+0000 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:15.582238+0000 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:15.583 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 1 Processor -- start 2024-03-25T18:29:15.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 1 -- start start 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient: init 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-b con 0x7f9d700b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-a con 0x7f9d700b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700af940 con 0x7f9d700b1730 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700b0a70 con 0x7f9d7014be90 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700b0d50 con 0x7f9d700b4d60 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:15.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.582+0000 7f9d75c96640 10 monclient(hunting): authenticate will time out at 8433.894803s 2024-03-25T18:29:15.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.587+0000 7f9d6f7fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f9d700b4d60 0x7f9d7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39994/0 (socket says 172.21.15.176:39994) 2024-03-25T18:29:15.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.587+0000 7f9d6f7fe640 1 -- 172.21.15.176:0/3349795444 learned_addr learned my addr 172.21.15.176:0/3349795444 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203970756 0 0) 0x7f9d700b0d50 con 0x7f9d700b4d60 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:29:15.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.599+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d58003520 con 0x7f9d700b4d60 2024-03-25T18:29:15.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674950711 0 0) 0x7f9d700b0a70 con 0x7f9d7014be90 2024-03-25T18:29:15.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:15.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:15.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:15.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:29:15.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.604+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d700b0d50 con 0x7f9d7014be90 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239516036 0 0) 0x7f9d58003520 con 0x7f9d700b4d60 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 >> v1:172.21.15.186:6789/0 conn(0x7f9d7014be90 legacy=0x7f9d7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 >> v1:172.21.15.176:6789/0 conn(0x7f9d700b1730 legacy=0x7f9d700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:29:15.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700b1030 con 0x7f9d700b4d60 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9d60001690 con 0x7f9d700b4d60 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d6e7fc640 10 monclient: dump: 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:15.618 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:15.619 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:15.619 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:15.619 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d75c96640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:29:15.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.616+0000 7f9d75c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d600017f0 con 0x7f9d700b4d60 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 1 -- 172.21.15.176:0/3349795444 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9d60001c00 con 0x7f9d700b4d60 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d75c96640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d75c96640 4 set_mon_vals no callback set 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d6e7fc640 10 monclient: dump: 2024-03-25T18:29:15.640 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:15.641 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d75c96640 10 monclient: shutdown 2024-03-25T18:29:15.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d75c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:15.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.639+0000 7f9d75c96640 1 -- 172.21.15.176:0/3349795444 >> v1:172.21.15.176:6790/0 conn(0x7f9d700b4d60 legacy=0x7f9d7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:15.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.640+0000 7f9d75c96640 1 -- 172.21.15.176:0/3349795444 shutdown_connections 2024-03-25T18:29:15.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.640+0000 7f9d75c96640 1 -- 172.21.15.176:0/3349795444 >> 172.21.15.176:0/3349795444 conn(0x7f9d700617a0 msgr2=0x7f9d70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:15.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.640+0000 7f9d75c96640 1 -- 172.21.15.176:0/3349795444 shutdown_connections 2024-03-25T18:29:15.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.640+0000 7f9d75c96640 1 -- 172.21.15.176:0/3349795444 wait complete. 2024-03-25T18:29:15.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 10 monclient: build_initial_monmap 2024-03-25T18:29:15.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 10 monclient: monmap: 2024-03-25T18:29:15.644 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 librados: starting msgr at 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 librados: starting objecter 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 Processor -- start 2024-03-25T18:29:15.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 -- start start 2024-03-25T18:29:15.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 librados: setting wanted keys 2024-03-25T18:29:15.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 1 librados: calling monclient init 2024-03-25T18:29:15.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.641+0000 7f9d75c96640 10 monclient: init 2024-03-25T18:29:15.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.642+0000 7f9d75c96640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:15.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.642+0000 7f9d75c96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.642+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-c con 0x7f9d700b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-b con 0x7f9d700b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 10 monclient(hunting): picked mon.noname-a con 0x7f9d7014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d701dbff0 con 0x7f9d7014be90 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d701327f0 con 0x7f9d700b4d60 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d70132f60 con 0x7f9d700b1730 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:15.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.643+0000 7f9d75c96640 10 monclient(hunting): authenticate will time out at 8433.955142s 2024-03-25T18:29:15.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.644+0000 7f9d6f7fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f9d700b1730 0x7f9d7012da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:40038/0 (socket says 172.21.15.176:40038) 2024-03-25T18:29:15.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.644+0000 7f9d6f7fe640 1 -- 172.21.15.176:0/1516621270 learned_addr learned my addr 172.21.15.176:0/1516621270 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:15.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.646+0000 7f9d57fff640 10 client.?.objecter ms_handle_connect 0x7f9d700b1730 2024-03-25T18:29:15.648 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.646+0000 7f9d57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:15.652 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.651+0000 7f9d57fff640 10 client.?.objecter ms_handle_connect 0x7f9d700b4d60 2024-03-25T18:29:15.652 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.651+0000 7f9d57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892289042 0 0) 0x7f9d70132f60 con 0x7f9d700b1730 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:29:15.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.657+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5c004eb0 con 0x7f9d700b1730 2024-03-25T18:29:15.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.665+0000 7f9d57fff640 10 client.?.objecter ms_handle_connect 0x7f9d7014be90 2024-03-25T18:29:15.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.665+0000 7f9d57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852504405 0 0) 0x7f9d701327f0 con 0x7f9d700b4d60 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:29:15.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.666+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d70132f60 con 0x7f9d700b4d60 2024-03-25T18:29:15.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753211252 0 0) 0x7f9d5c004eb0 con 0x7f9d700b1730 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d701327f0 con 0x7f9d700b1730 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9d60001bc0 con 0x7f9d700b1730 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 10 monclient(hunting): dump: 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:15.673 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.671+0000 7f9d57fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2168442867 0 0) 0x7f9d70132f60 con 0x7f9d700b4d60 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d5c004eb0 con 0x7f9d700b4d60 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9d640018b0 con 0x7f9d700b4d60 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.684+0000 7f9d57fff640 10 monclient(hunting): dump: 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:15.686 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312685452 0 0) 0x7f9d701dbff0 con 0x7f9d7014be90 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:15.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:15.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:15.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:29:15.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.686+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d70132f60 con 0x7f9d7014be90 2024-03-25T18:29:15.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2090957129 0 0) 0x7f9d701327f0 con 0x7f9d700b1730 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 >> v1:172.21.15.186:6789/0 conn(0x7f9d700b4d60 legacy=0x7f9d70137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 >> v1:172.21.15.176:6789/0 conn(0x7f9d7014be90 legacy=0x7f9d701e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 1 monclient: found mon.c 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7012f5b0 con 0x7f9d700b1730 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 10 monclient: _renew_subs 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:15.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7012f900 con 0x7f9d700b1730 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 10 monclient: _renew_subs 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d701db090 con 0x7f9d700b1730 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.696+0000 7f9d75c96640 1 librados: init done 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.697+0000 7f9d75c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:15.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.697+0000 7f9d75c96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:15.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.697+0000 7f9d75c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:15.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.697+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d70132bb0 con 0x7f9d700b1730 2024-03-25T18:29:15.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.916+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d600011f0 con 0x7f9d700b1730 2024-03-25T18:29:15.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.916+0000 7f9d57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:15.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:15.916+0000 7f9d6dffb640 4 set_mon_vals no callback set 2024-03-25T18:29:16.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.006+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9d60004cb0 con 0x7f9d700b1730 2024-03-25T18:29:16.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.006+0000 7f9d57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:16.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.006+0000 7f9d57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:16.007 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.006+0000 7f9d57fff640 10 monclient: dump: 2024-03-25T18:29:16.007 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:16.008 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:16.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.007+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f9d6001ad40 con 0x7f9d700b1730 2024-03-25T18:29:16.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.007+0000 7f9d6effd640 10 monclient: get_auth_request con 0x7f9d5c06deb0 auth_method 0 2024-03-25T18:29:16.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.008+0000 7f9d57fff640 10 client.4454.objecter ms_handle_connect 0x7f9d5c06deb0 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f9d60054a00 con 0x7f9d700b1730 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 10 client.4454.objecter ms_dispatch 0x7f9d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d60050340 con 0x7f9d700b1730 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:16.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.014+0000 7f9d57fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:16.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.266+0000 7f9d75c96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:16.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.266+0000 7f9d75c96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:16.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.266+0000 7f9d75c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:16.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.266+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d700627b0 con 0x7f9d700b1730 2024-03-25T18:29:16.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.301+0000 7f9d57fff640 1 -- 172.21.15.176:0/1516621270 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f9d60052440 con 0x7f9d700b1730 2024-03-25T18:29:16.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.301+0000 7f9d57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:16.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.301+0000 7f9d57fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:16.303 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:16.303 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 10 librados: watch_flush enter 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 10 librados: watch_flush exit 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 10 client.4454.objecter successfully canceled tick 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 >> v1:172.21.15.176:6800/2598213983 conn(0x7f9d5c06deb0 legacy=0x7f9d5c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 10 monclient: shutdown 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 >> v1:172.21.15.176:6790/0 conn(0x7f9d700b1730 legacy=0x7f9d7012da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 shutdown_connections 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 >> 172.21.15.176:0/1516621270 conn(0x7f9d700617a0 msgr2=0x7f9d700b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 shutdown_connections 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 -- 172.21.15.176:0/1516621270 wait complete. 2024-03-25T18:29:16.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:16.304+0000 7f9d75c96640 1 librados: shutdown 2024-03-25T18:29:16.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:23.123 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.237+0000 7efc30093640 10 monclient: get_monmap_and_config 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.237+0000 7efc30093640 10 monclient: build_initial_monmap 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.237+0000 7efc30093640 10 monclient: monmap: 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:23.238 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:23.238257+0000 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:23.238257+0000 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:23.239 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 1 Processor -- start 2024-03-25T18:29:23.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 1 -- start start 2024-03-25T18:29:23.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient: init 2024-03-25T18:29:23.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:23.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient(hunting): picked mon.noname-a con 0x7efc280a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient(hunting): picked mon.noname-c con 0x7efc2813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 10 monclient(hunting): picked mon.noname-b con 0x7efc281430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc28142520 con 0x7efc280a98f0 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.238+0000 7efc30093640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc280b28a0 con 0x7efc2813f710 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.239+0000 7efc30093640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc280b2a30 con 0x7efc281430c0 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.239+0000 7efc30093640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:23.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.239+0000 7efc30093640 10 monclient(hunting): authenticate will time out at 8441.550858s 2024-03-25T18:29:23.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.244+0000 7efc2de2f640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7efc280a98f0 0x7efc280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:52676/0 (socket says 172.21.15.176:52676) 2024-03-25T18:29:23.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.245+0000 7efc2de2f640 1 -- 172.21.15.176:0/839061360 learned_addr learned my addr 172.21.15.176:0/839061360 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:23.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087329016 0 0) 0x7efc280b28a0 con 0x7efc2813f710 2024-03-25T18:29:23.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 10 monclient(hunting): _negotiate 2024-03-25T18:29:23.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:23.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:23.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:29:23.256 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.254+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc14003520 con 0x7efc2813f710 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038100335 0 0) 0x7efc280b2a30 con 0x7efc281430c0 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 10 monclient(hunting): _negotiate 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:29:23.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.262+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc280b28a0 con 0x7efc281430c0 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.264+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3524821884 0 0) 0x7efc14003520 con 0x7efc2813f710 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 >> v1:172.21.15.176:6790/0 conn(0x7efc281430c0 legacy=0x7efc2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 >> v1:172.21.15.176:6789/0 conn(0x7efc280a98f0 legacy=0x7efc280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: _finish_hunting 0 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 1 monclient: found mon.noname-c 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:29:23.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc280b44c0 con 0x7efc2813f710 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: _finish_auth 0 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: _check_auth_tickets 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efc18001690 con 0x7efc2813f710 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc2ce2d640 10 monclient: dump: 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:23.267 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:23.268 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:23.268 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc30093640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:29:23.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.265+0000 7efc30093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:23.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc180017f0 con 0x7efc2813f710 2024-03-25T18:29:23.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:23.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 1 -- 172.21.15.176:0/839061360 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efc18001c00 con 0x7efc2813f710 2024-03-25T18:29:23.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc30093640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:23.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc30093640 4 set_mon_vals no callback set 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.293+0000 7efc2ce2d640 10 monclient: dump: 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:23.295 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.294+0000 7efc30093640 10 monclient: shutdown 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.294+0000 7efc30093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.294+0000 7efc30093640 1 -- 172.21.15.176:0/839061360 >> v1:172.21.15.186:6789/0 conn(0x7efc2813f710 legacy=0x7efc281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.294+0000 7efc30093640 1 -- 172.21.15.176:0/839061360 shutdown_connections 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.294+0000 7efc30093640 1 -- 172.21.15.176:0/839061360 >> 172.21.15.176:0/839061360 conn(0x7efc280b7ea0 msgr2=0x7efc280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.295+0000 7efc30093640 1 -- 172.21.15.176:0/839061360 shutdown_connections 2024-03-25T18:29:23.296 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.295+0000 7efc30093640 1 -- 172.21.15.176:0/839061360 wait complete. 2024-03-25T18:29:23.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 10 monclient: build_initial_monmap 2024-03-25T18:29:23.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 10 monclient: monmap: 2024-03-25T18:29:23.297 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:23.297 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 librados: starting msgr at 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 librados: starting objecter 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 Processor -- start 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 -- start start 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 librados: setting wanted keys 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 1 librados: calling monclient init 2024-03-25T18:29:23.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.296+0000 7efc30093640 10 monclient: init 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient(hunting): picked mon.noname-b con 0x7efc280a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient(hunting): picked mon.noname-a con 0x7efc2813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient(hunting): picked mon.noname-c con 0x7efc281430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc281db660 con 0x7efc2813f710 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc281db7f0 con 0x7efc280a98f0 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2814dee0 con 0x7efc281430c0 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:23.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.297+0000 7efc30093640 10 monclient(hunting): authenticate will time out at 8441.609462s 2024-03-25T18:29:23.304 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.303+0000 7efc2e630640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7efc281430c0 0x7efc281e1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:44840/0 (socket says 172.21.15.176:44840) 2024-03-25T18:29:23.304 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.303+0000 7efc2e630640 1 -- 172.21.15.176:0/3239688346 learned_addr learned my addr 172.21.15.176:0/3239688346 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:23.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.305+0000 7efc1e7fc640 10 client.?.objecter ms_handle_connect 0x7efc281430c0 2024-03-25T18:29:23.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.305+0000 7efc1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:23.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.308+0000 7efc1e7fc640 10 client.?.objecter ms_handle_connect 0x7efc2813f710 2024-03-25T18:29:23.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.308+0000 7efc1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.315+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864213378 0 0) 0x7efc281db660 con 0x7efc2813f710 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.315+0000 7efc1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.315+0000 7efc1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.315+0000 7efc1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.316+0000 7efc1e7fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:29:23.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.316+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc0c004eb0 con 0x7efc2813f710 2024-03-25T18:29:23.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.320+0000 7efc1e7fc640 10 client.?.objecter ms_handle_connect 0x7efc280a98f0 2024-03-25T18:29:23.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.320+0000 7efc1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015721712 0 0) 0x7efc0c004eb0 con 0x7efc2813f710 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc281db660 con 0x7efc2813f710 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efc18001bc0 con 0x7efc2813f710 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:23.328 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.327+0000 7efc1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:23.329 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812086212 0 0) 0x7efc2814dee0 con 0x7efc281430c0 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.328+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc0c004eb0 con 0x7efc281430c0 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545504234 0 0) 0x7efc281db7f0 con 0x7efc280a98f0 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:23.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:23.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:23.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:29:23.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.329+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2814dee0 con 0x7efc280a98f0 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.342+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3234828869 0 0) 0x7efc281db660 con 0x7efc2813f710 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 >> v1:172.21.15.176:6790/0 conn(0x7efc281430c0 legacy=0x7efc281e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 >> v1:172.21.15.186:6789/0 conn(0x7efc280a98f0 legacy=0x7efc281dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 1 monclient: found mon.a 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:29:23.344 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2814e070 con 0x7efc2813f710 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 10 monclient: _renew_subs 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc280b11e0 con 0x7efc2813f710 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 10 monclient: _renew_subs 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc280b1710 con 0x7efc2813f710 2024-03-25T18:29:23.345 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.343+0000 7efc30093640 1 librados: init done 2024-03-25T18:29:23.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.344+0000 7efc30093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:23.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.344+0000 7efc30093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:23.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.344+0000 7efc30093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:29:23.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.344+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2814e260 con 0x7efc2813f710 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc180038a0 con 0x7efc2813f710 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efc18003bb0 con 0x7efc2813f710 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:23.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1e7fc640 10 monclient: dump: 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:23.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.670+0000 7efc1ffff640 4 set_mon_vals no callback set 2024-03-25T18:29:23.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.671+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7efc18019b80 con 0x7efc2813f710 2024-03-25T18:29:23.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.672+0000 7efc2de2f640 10 monclient: get_auth_request con 0x7efc0c06de80 auth_method 0 2024-03-25T18:29:23.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.672+0000 7efc1e7fc640 10 client.4437.objecter ms_handle_connect 0x7efc0c06de80 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7efc18053740 con 0x7efc2813f710 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 10 client.4437.objecter ms_dispatch 0x7efc28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc1804f080 con 0x7efc2813f710 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:23.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.677+0000 7efc1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:23.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.926+0000 7efc30093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:23.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.926+0000 7efc30093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:23.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.926+0000 7efc30093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:29:23.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.926+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc280627b0 con 0x7efc2813f710 2024-03-25T18:29:23.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.957+0000 7efc1e7fc640 1 -- 172.21.15.176:0/3239688346 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7efc18051180 con 0x7efc2813f710 2024-03-25T18:29:23.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.958+0000 7efc1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:23.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.958+0000 7efc1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:23.959 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:23.959 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 10 librados: watch_flush enter 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 10 librados: watch_flush exit 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 10 client.4437.objecter successfully canceled tick 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 >> v1:172.21.15.176:6800/2598213983 conn(0x7efc0c06de80 legacy=0x7efc0c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 10 monclient: shutdown 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 >> v1:172.21.15.176:6789/0 conn(0x7efc2813f710 legacy=0x7efc280ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 shutdown_connections 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 >> 172.21.15.176:0/3239688346 conn(0x7efc280b7ea0 msgr2=0x7efc28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 shutdown_connections 2024-03-25T18:29:23.962 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.960+0000 7efc30093640 1 -- 172.21.15.176:0/3239688346 wait complete. 2024-03-25T18:29:23.963 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:23.961+0000 7efc30093640 1 librados: shutdown 2024-03-25T18:29:23.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:30.779 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:30.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.891+0000 7f7fa8efb640 10 monclient: get_monmap_and_config 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.891+0000 7f7fa8efb640 10 monclient: build_initial_monmap 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.891+0000 7f7fa8efb640 10 monclient: monmap: 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:30.892690+0000 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:30.892690+0000 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:30.893 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.892+0000 7f7fa8efb640 1 Processor -- start 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.892+0000 7f7fa8efb640 1 -- start start 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.892+0000 7f7fa8efb640 10 monclient: init 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa4142520 con 0x7f7fa413f710 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa40b28a0 con 0x7f7fa41430c0 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa40b2a30 con 0x7f7fa40a98f0 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:30.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.893+0000 7f7fa8efb640 10 monclient(hunting): authenticate will time out at 8449.205419s 2024-03-25T18:29:30.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.903+0000 7f7fa1d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7fa413f710 0x7f7fa41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57682/0 (socket says 172.21.15.176:57682) 2024-03-25T18:29:30.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.903+0000 7f7fa1d9b640 1 -- 172.21.15.176:0/1744431967 learned_addr learned my addr 172.21.15.176:0/1744431967 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332002429 0 0) 0x7f7fa4142520 con 0x7f7fa413f710 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:29:30.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.916+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f8c003520 con 0x7f7fa413f710 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680294680 0 0) 0x7f7fa40b28a0 con 0x7f7fa41430c0 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:29:30.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.918+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa4142520 con 0x7f7fa41430c0 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098966357 0 0) 0x7f7f8c003520 con 0x7f7fa413f710 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 >> v1:172.21.15.176:6790/0 conn(0x7f7fa40a98f0 legacy=0x7f7fa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 >> v1:172.21.15.186:6789/0 conn(0x7f7fa41430c0 legacy=0x7f7fa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 monclient: found mon.noname-a 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40b44c0 con 0x7f7fa413f710 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: _finish_auth 0 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:30.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7f98001690 con 0x7f7fa413f710 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa159a640 10 monclient: dump: 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa8efb640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:29:30.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.929+0000 7f7fa8efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f980017f0 con 0x7f7fa413f710 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 1 -- 172.21.15.176:0/1744431967 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7f98001c00 con 0x7f7fa413f710 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa8efb640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa8efb640 4 set_mon_vals no callback set 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:30.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.936+0000 7f7fa159a640 10 monclient: dump: 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:30.938 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 10 monclient: shutdown 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 1 -- 172.21.15.176:0/1744431967 >> v1:172.21.15.176:6789/0 conn(0x7f7fa413f710 legacy=0x7f7fa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 1 -- 172.21.15.176:0/1744431967 shutdown_connections 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 1 -- 172.21.15.176:0/1744431967 >> 172.21.15.176:0/1744431967 conn(0x7f7fa40b7ea0 msgr2=0x7f7fa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 1 -- 172.21.15.176:0/1744431967 shutdown_connections 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.937+0000 7f7fa8efb640 1 -- 172.21.15.176:0/1744431967 wait complete. 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.938+0000 7f7fa8efb640 10 monclient: build_initial_monmap 2024-03-25T18:29:30.940 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.938+0000 7f7fa8efb640 10 monclient: monmap: 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 librados: starting msgr at 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 librados: starting objecter 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 Processor -- start 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 -- start start 2024-03-25T18:29:30.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 librados: setting wanted keys 2024-03-25T18:29:30.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 librados: calling monclient init 2024-03-25T18:29:30.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient: init 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa40a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.939+0000 7f7fa8efb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa40abd70 con 0x7f7fa41430c0 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.940+0000 7f7fa8efb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa41d7710 con 0x7f7fa40a98f0 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.940+0000 7f7fa8efb640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa41d8340 con 0x7f7fa413f710 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.940+0000 7f7fa8efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:30.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.940+0000 7f7fa8efb640 10 monclient(hunting): authenticate will time out at 8449.251881s 2024-03-25T18:29:30.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.942+0000 7f7fa1d9b640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f7fa413f710 0x7f7fa41d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:38148/0 (socket says 172.21.15.176:38148) 2024-03-25T18:29:30.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.942+0000 7f7fa1d9b640 1 -- 172.21.15.176:0/4115217222 learned_addr learned my addr 172.21.15.176:0/4115217222 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:30.946 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.945+0000 7f7f92ffd640 10 client.?.objecter ms_handle_connect 0x7f7fa413f710 2024-03-25T18:29:30.946 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.945+0000 7f7f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:30.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.950+0000 7f7f92ffd640 10 client.?.objecter ms_handle_connect 0x7f7fa40a98f0 2024-03-25T18:29:30.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.950+0000 7f7f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:30.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943240594 0 0) 0x7f7fa41d8340 con 0x7f7fa413f710 2024-03-25T18:29:30.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:30.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:30.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:30.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:29:30.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.956+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f84004eb0 con 0x7f7fa413f710 2024-03-25T18:29:30.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.959+0000 7f7f92ffd640 10 client.?.objecter ms_handle_connect 0x7f7fa41430c0 2024-03-25T18:29:30.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.959+0000 7f7f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:30.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17163694 0 0) 0x7f7fa41d7710 con 0x7f7fa40a98f0 2024-03-25T18:29:30.972 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:30.972 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:30.972 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:30.972 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:29:30.972 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.970+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa41d8340 con 0x7f7fa40a98f0 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189220278 0 0) 0x7f7fa40abd70 con 0x7f7fa41430c0 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.972+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa41d7710 con 0x7f7fa41430c0 2024-03-25T18:29:30.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1702862927 0 0) 0x7f7f84004eb0 con 0x7f7fa413f710 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa40abd70 con 0x7f7fa413f710 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7f98001bc0 con 0x7f7fa413f710 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:30.975 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.973+0000 7f7f92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:30.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357119877 0 0) 0x7f7fa41d8340 con 0x7f7fa40a98f0 2024-03-25T18:29:30.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f84004eb0 con 0x7f7fa40a98f0 2024-03-25T18:29:30.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7f940018b0 con 0x7f7fa40a98f0 2024-03-25T18:29:30.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:30.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.983+0000 7f7f92ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:30.985 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:30.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 748496948 0 0) 0x7f7fa40abd70 con 0x7f7fa413f710 2024-03-25T18:29:30.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 >> v1:172.21.15.186:6789/0 conn(0x7f7fa40a98f0 legacy=0x7f7fa4137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 >> v1:172.21.15.176:6789/0 conn(0x7f7fa41430c0 legacy=0x7f7fa41d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 1 monclient: found mon.c 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa41d84d0 con 0x7f7fa413f710 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 10 monclient: _renew_subs 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:30.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa41d41e0 con 0x7f7fa413f710 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 10 monclient: _renew_subs 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa41d5150 con 0x7f7fa413f710 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.986+0000 7f7fa8efb640 1 librados: init done 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.987+0000 7f7fa8efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:30.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.987+0000 7f7fa8efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:30.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.987+0000 7f7fa8efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:30.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:30.987+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa40627b0 con 0x7f7fa413f710 2024-03-25T18:29:31.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f980011f0 con 0x7f7fa413f710 2024-03-25T18:29:31.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:31.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7f98004c30 con 0x7f7fa413f710 2024-03-25T18:29:31.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:31.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7f92ffd640 10 monclient: dump: 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:31.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.309+0000 7f7fa0d99640 4 set_mon_vals no callback set 2024-03-25T18:29:31.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.310+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f7f9801ac80 con 0x7f7fa413f710 2024-03-25T18:29:31.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.311+0000 7f7fa259c640 10 monclient: get_auth_request con 0x7f7f8406df20 auth_method 0 2024-03-25T18:29:31.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.311+0000 7f7f92ffd640 10 client.4466.objecter ms_handle_connect 0x7f7f8406df20 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7f98054930 con 0x7f7fa413f710 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 10 client.4466.objecter ms_dispatch 0x7f7fa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f98050270 con 0x7f7fa413f710 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:31.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.315+0000 7f7f92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:31.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.560+0000 7f7fa8efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:31.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.560+0000 7f7fa8efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:31.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.560+0000 7f7fa8efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:31.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.560+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa4074cc0 con 0x7f7fa413f710 2024-03-25T18:29:31.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.589+0000 7f7f92ffd640 1 -- 172.21.15.176:0/4115217222 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7f98052370 con 0x7f7fa413f710 2024-03-25T18:29:31.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.589+0000 7f7f92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:31.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.589+0000 7f7f92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:31.591 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:31.591 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 10 librados: watch_flush enter 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 10 librados: watch_flush exit 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 >> v1:172.21.15.176:6800/2598213983 conn(0x7f7f8406df20 legacy=0x7f7f840703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 10 monclient: shutdown 2024-03-25T18:29:31.594 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 >> v1:172.21.15.176:6790/0 conn(0x7f7fa413f710 legacy=0x7f7fa41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 shutdown_connections 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 >> 172.21.15.176:0/4115217222 conn(0x7f7fa40b7ea0 msgr2=0x7f7fa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 shutdown_connections 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.592+0000 7f7fa8efb640 1 -- 172.21.15.176:0/4115217222 wait complete. 2024-03-25T18:29:31.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:31.593+0000 7f7fa8efb640 1 librados: shutdown 2024-03-25T18:29:31.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:33.366 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:35.122 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.409 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:35.165 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.412 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:35.146 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.413 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:35.147 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.431 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:35.187 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.448 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:35.204 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.461 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:35.195 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.489 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:35.245 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.505 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:35.261 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:38.412 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:38.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.529+0000 7fccccb67640 10 monclient: get_monmap_and_config 2024-03-25T18:29:38.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.529+0000 7fccccb67640 10 monclient: build_initial_monmap 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.529+0000 7fccccb67640 10 monclient: monmap: 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:38.530449+0000 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:38.530449+0000 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:38.531 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.530+0000 7fccccb67640 1 Processor -- start 2024-03-25T18:29:38.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.530+0000 7fccccb67640 1 -- start start 2024-03-25T18:29:38.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.530+0000 7fccccb67640 10 monclient: init 2024-03-25T18:29:38.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-a con 0x7fccc80a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-b con 0x7fccc813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-c con 0x7fccc81430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc8142520 con 0x7fccc80a98f0 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80b28a0 con 0x7fccc81430c0 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80b2a30 con 0x7fccc813f710 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:38.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.531+0000 7fccccb67640 10 monclient(hunting): authenticate will time out at 8456.843319s 2024-03-25T18:29:38.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.543+0000 7fccc659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fccc80a98f0 0x7fccc80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:41434/0 (socket says 172.21.15.176:41434) 2024-03-25T18:29:38.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.543+0000 7fccc659c640 1 -- 172.21.15.176:0/354611123 learned_addr learned my addr 172.21.15.176:0/354611123 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475035234 0 0) 0x7fccc80b28a0 con 0x7fccc81430c0 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:29:38.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.554+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb0003520 con 0x7fccc81430c0 2024-03-25T18:29:38.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621801710 0 0) 0x7fccc80b2a30 con 0x7fccc813f710 2024-03-25T18:29:38.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:29:38.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.557+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc80b28a0 con 0x7fccc813f710 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927371052 0 0) 0x7fccc8142520 con 0x7fccc80a98f0 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:29:38.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.561+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc80b2a30 con 0x7fccc80a98f0 2024-03-25T18:29:38.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3961729570 0 0) 0x7fccc80b28a0 con 0x7fccc813f710 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 >> v1:172.21.15.186:6789/0 conn(0x7fccc81430c0 legacy=0x7fccc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 >> v1:172.21.15.176:6789/0 conn(0x7fccc80a98f0 legacy=0x7fccc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 monclient: found mon.noname-b 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80b44c0 con 0x7fccc813f710 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: _finish_auth 0 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fccbc001670 con 0x7fccc813f710 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:38.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccc559a640 10 monclient: dump: 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccccb67640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:29:38.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.573+0000 7fccccb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.597+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccbc0017d0 con 0x7fccc813f710 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccc559a640 1 -- 172.21.15.176:0/354611123 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fccbc001be0 con 0x7fccc813f710 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccc559a640 10 monclient: dump: 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.599 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccccb67640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:38.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccccb67640 4 set_mon_vals no callback set 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccccb67640 10 monclient: shutdown 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccccb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.598+0000 7fccccb67640 1 -- 172.21.15.176:0/354611123 >> v1:172.21.15.176:6790/0 conn(0x7fccc813f710 legacy=0x7fccc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.599+0000 7fccccb67640 1 -- 172.21.15.176:0/354611123 shutdown_connections 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.599+0000 7fccccb67640 1 -- 172.21.15.176:0/354611123 >> 172.21.15.176:0/354611123 conn(0x7fccc80b7ea0 msgr2=0x7fccc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.599+0000 7fccccb67640 1 -- 172.21.15.176:0/354611123 shutdown_connections 2024-03-25T18:29:38.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.599+0000 7fccccb67640 1 -- 172.21.15.176:0/354611123 wait complete. 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 10 monclient: build_initial_monmap 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 10 monclient: monmap: 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 librados: starting msgr at 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 librados: starting objecter 2024-03-25T18:29:38.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 Processor -- start 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 -- start start 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 librados: setting wanted keys 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 1 librados: calling monclient init 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.600+0000 7fccccb67640 10 monclient: init 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-b con 0x7fccc80a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-a con 0x7fccc813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient(hunting): picked mon.noname-c con 0x7fccc81430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc8136fa0 con 0x7fccc813f710 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc813b780 con 0x7fccc80a98f0 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc81d7ba0 con 0x7fccc81430c0 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:38.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.601+0000 7fccccb67640 10 monclient(hunting): authenticate will time out at 8456.913161s 2024-03-25T18:29:38.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.610+0000 7fccc659c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fccc80a98f0 0x7fccc8136890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:45098/0 (socket says 172.21.15.176:45098) 2024-03-25T18:29:38.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.610+0000 7fccc659c640 1 -- 172.21.15.176:0/724017493 learned_addr learned my addr 172.21.15.176:0/724017493 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:38.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.611+0000 7fccb6ffd640 10 client.?.objecter ms_handle_connect 0x7fccc80a98f0 2024-03-25T18:29:38.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.611+0000 7fccb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:38.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 client.?.objecter ms_handle_connect 0x7fccc81430c0 2024-03-25T18:29:38.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:38.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999804906 0 0) 0x7fccc813b780 con 0x7fccc80a98f0 2024-03-25T18:29:38.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:29:38.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.625+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca8004eb0 con 0x7fccc80a98f0 2024-03-25T18:29:38.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.631+0000 7fccb6ffd640 10 client.?.objecter ms_handle_connect 0x7fccc813f710 2024-03-25T18:29:38.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.631+0000 7fccb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:38.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500402879 0 0) 0x7fccc81d7ba0 con 0x7fccc81430c0 2024-03-25T18:29:38.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc813b780 con 0x7fccc81430c0 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2113761445 0 0) 0x7fcca8004eb0 con 0x7fccc80a98f0 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc81d7ba0 con 0x7fccc80a98f0 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fccb8001910 con 0x7fccc80a98f0 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.641 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.639+0000 7fccb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206433837 0 0) 0x7fccc8136fa0 con 0x7fccc813f710 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:29:38.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.646+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca8004eb0 con 0x7fccc813f710 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059712050 0 0) 0x7fccc813b780 con 0x7fccc81430c0 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc8136fa0 con 0x7fccc81430c0 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fccc0001a70 con 0x7fccc81430c0 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.652+0000 7fccb6ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:38.653 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:38.654 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 343284792 0 0) 0x7fccc81d7ba0 con 0x7fccc80a98f0 2024-03-25T18:29:38.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 >> v1:172.21.15.176:6790/0 conn(0x7fccc81430c0 legacy=0x7fccc81e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 >> v1:172.21.15.176:6789/0 conn(0x7fccc813f710 legacy=0x7fccc81d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 1 monclient: found mon.b 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8137990 con 0x7fccc80a98f0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 10 monclient: _renew_subs 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc8137c40 con 0x7fccc80a98f0 2024-03-25T18:29:38.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 10 monclient: _renew_subs 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc8139860 con 0x7fccc80a98f0 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.655+0000 7fccccb67640 1 librados: init done 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.656+0000 7fccccb67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.656+0000 7fccccb67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:38.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.656+0000 7fccccb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:38.659 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.656+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc80627b0 con 0x7fccc80a98f0 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb80047e0 con 0x7fccc80a98f0 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fccb8004bf0 con 0x7fccc80a98f0 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccb6ffd640 10 monclient: dump: 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.951 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.949+0000 7fccc4d99640 4 set_mon_vals no callback set 2024-03-25T18:29:38.952 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.950+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fccb8004e10 con 0x7fccc80a98f0 2024-03-25T18:29:38.953 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.951+0000 7fccc5d9b640 10 monclient: get_auth_request con 0x7fcca806df10 auth_method 0 2024-03-25T18:29:38.953 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.951+0000 7fccb6ffd640 10 client.4462.objecter ms_handle_connect 0x7fcca806df10 2024-03-25T18:29:38.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fccb8054840 con 0x7fccc80a98f0 2024-03-25T18:29:38.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 10 client.4462.objecter ms_dispatch 0x7fccc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:38.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:38.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:38.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:29:38.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccb8050350 con 0x7fccc80a98f0 2024-03-25T18:29:38.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:38.959 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:38.957+0000 7fccb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:39.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.206+0000 7fccccb67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:39.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.206+0000 7fccccb67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:39.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.206+0000 7fccccb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:29:39.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.206+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc8069f40 con 0x7fccc80a98f0 2024-03-25T18:29:39.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.231+0000 7fccb6ffd640 1 -- 172.21.15.176:0/724017493 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fccb8052280 con 0x7fccc80a98f0 2024-03-25T18:29:39.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.231+0000 7fccb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:39.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.231+0000 7fccb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:39.233 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:39.233 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:39.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 10 librados: watch_flush enter 2024-03-25T18:29:39.235 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 10 librados: watch_flush exit 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 >> v1:172.21.15.176:6800/2598213983 conn(0x7fcca806df10 legacy=0x7fcca80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 10 monclient: shutdown 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 >> v1:172.21.15.186:6789/0 conn(0x7fccc80a98f0 legacy=0x7fccc8136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 shutdown_connections 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 >> 172.21.15.176:0/724017493 conn(0x7fccc80b7ea0 msgr2=0x7fccc814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 shutdown_connections 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 -- 172.21.15.176:0/724017493 wait complete. 2024-03-25T18:29:39.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:39.234+0000 7fccccb67640 1 librados: shutdown 2024-03-25T18:29:39.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:46.056 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.171+0000 7f036c7f4640 10 monclient: get_monmap_and_config 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.171+0000 7f036c7f4640 10 monclient: build_initial_monmap 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.171+0000 7f036c7f4640 10 monclient: monmap: 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:46.172248+0000 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:46.172248+0000 2024-03-25T18:29:46.175 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.171+0000 7f036c7f4640 1 Processor -- start 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 1 -- start start 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient: init 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f03640a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f036413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f03641430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0364142520 con 0x7f03641430c0 2024-03-25T18:29:46.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640b28a0 con 0x7f036413f710 2024-03-25T18:29:46.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640b2a30 con 0x7f03640a98f0 2024-03-25T18:29:46.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:46.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.172+0000 7f036c7f4640 10 monclient(hunting): authenticate will time out at 8464.484792s 2024-03-25T18:29:46.178 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.178+0000 7f0369d8f640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f036413f710 0x7f03641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:44566/0 (socket says 172.21.15.176:44566) 2024-03-25T18:29:46.178 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.178+0000 7f0369d8f640 1 -- 172.21.15.176:0/3428856119 learned_addr learned my addr 172.21.15.176:0/3428856119 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091747415 0 0) 0x7f03640b28a0 con 0x7f036413f710 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:29:46.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.188+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0350003520 con 0x7f036413f710 2024-03-25T18:29:46.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956427259 0 0) 0x7f0364142520 con 0x7f03641430c0 2024-03-25T18:29:46.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:29:46.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.199+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03640b28a0 con 0x7f03641430c0 2024-03-25T18:29:46.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600565435 0 0) 0x7f03640b2a30 con 0x7f03640a98f0 2024-03-25T18:29:46.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:29:46.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.200+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0364142520 con 0x7f03640a98f0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343933097 0 0) 0x7f03640b28a0 con 0x7f03641430c0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 >> v1:172.21.15.176:6790/0 conn(0x7f03640a98f0 legacy=0x7f03640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 >> v1:172.21.15.186:6789/0 conn(0x7f036413f710 legacy=0x7f03641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: _finish_hunting 0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 monclient: found mon.noname-a 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03640b44c0 con 0x7f03641430c0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: _finish_auth 0 2024-03-25T18:29:46.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: _check_auth_tickets 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f035c0019f0 con 0x7f03641430c0 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036958e640 10 monclient: dump: 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:46.208 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:46.209 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.205+0000 7f036c7f4640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:29:46.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.206+0000 7f036c7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f035c001b50 con 0x7f03641430c0 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 1 -- 172.21.15.176:0/3428856119 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f035c001f60 con 0x7f03641430c0 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036958e640 10 monclient: dump: 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.210 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:46.211 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:46.211 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:46.211 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036c7f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:46.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.208+0000 7f036c7f4640 4 set_mon_vals no callback set 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.209+0000 7f036c7f4640 10 monclient: shutdown 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.209+0000 7f036c7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.209+0000 7f036c7f4640 1 -- 172.21.15.176:0/3428856119 >> v1:172.21.15.176:6789/0 conn(0x7f03641430c0 legacy=0x7f036414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 1 -- 172.21.15.176:0/3428856119 shutdown_connections 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 1 -- 172.21.15.176:0/3428856119 >> 172.21.15.176:0/3428856119 conn(0x7f03640b7ea0 msgr2=0x7f03640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 1 -- 172.21.15.176:0/3428856119 shutdown_connections 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 1 -- 172.21.15.176:0/3428856119 wait complete. 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 10 monclient: build_initial_monmap 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.210+0000 7f036c7f4640 10 monclient: monmap: 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:46.213 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 librados: starting msgr at 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 librados: starting objecter 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 Processor -- start 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 -- start start 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 librados: setting wanted keys 2024-03-25T18:29:46.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 1 librados: calling monclient init 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 10 monclient: init 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.211+0000 7f036c7f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f03640a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f036413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f03641430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0364135bf0 con 0x7f03641430c0 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640b6d10 con 0x7f036413f710 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03641317f0 con 0x7f03640a98f0 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:46.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.212+0000 7f036c7f4640 10 monclient(hunting): authenticate will time out at 8464.523922s 2024-03-25T18:29:46.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.225+0000 7f036a590640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f03640a98f0 0x7f03641310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:58170/0 (socket says 172.21.15.176:58170) 2024-03-25T18:29:46.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.226+0000 7f036a590640 1 -- 172.21.15.176:0/1004423284 learned_addr learned my addr 172.21.15.176:0/1004423284 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:46.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.227+0000 7f035affd640 10 client.?.objecter ms_handle_connect 0x7f03640a98f0 2024-03-25T18:29:46.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.227+0000 7f035affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:46.231 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.230+0000 7f035affd640 10 client.?.objecter ms_handle_connect 0x7f036413f710 2024-03-25T18:29:46.231 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.230+0000 7f035affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:46.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105746905 0 0) 0x7f03641317f0 con 0x7f03640a98f0 2024-03-25T18:29:46.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:29:46.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.239+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0348004eb0 con 0x7f03640a98f0 2024-03-25T18:29:46.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.240+0000 7f035affd640 10 client.?.objecter ms_handle_connect 0x7f03641430c0 2024-03-25T18:29:46.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.240+0000 7f035affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502272715 0 0) 0x7f03640b6d10 con 0x7f036413f710 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:29:46.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.245+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03641317f0 con 0x7f036413f710 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157344055 0 0) 0x7f0364135bf0 con 0x7f03641430c0 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:29:46.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.249+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03640b6d10 con 0x7f03641430c0 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813206302 0 0) 0x7f0348004eb0 con 0x7f03640a98f0 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0364135bf0 con 0x7f03640a98f0 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0360001a10 con 0x7f03640a98f0 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 10 monclient(hunting): dump: 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.252 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:46.253 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:46.253 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:46.253 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.251+0000 7f035affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175766486 0 0) 0x7f03640b6d10 con 0x7f03641430c0 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0348004eb0 con 0x7f03641430c0 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f035c001e60 con 0x7f03641430c0 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 10 monclient(hunting): dump: 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.262 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2229796172 0 0) 0x7f0364135bf0 con 0x7f03640a98f0 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.260+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 >> v1:172.21.15.186:6789/0 conn(0x7f036413f710 legacy=0x7f03641dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 >> v1:172.21.15.176:6789/0 conn(0x7f03641430c0 legacy=0x7f03641db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 1 monclient: found mon.c 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0364131980 con 0x7f03640a98f0 2024-03-25T18:29:46.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 10 monclient: _finish_auth 0 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f035affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 10 monclient: _renew_subs 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03640b76e0 con 0x7f03640a98f0 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 10 monclient: _renew_subs 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0364135270 con 0x7f03640a98f0 2024-03-25T18:29:46.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.261+0000 7f036c7f4640 1 librados: init done 2024-03-25T18:29:46.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.262+0000 7f036c7f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:46.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.262+0000 7f036c7f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:46.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.262+0000 7f036c7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:46.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.262+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03640627b0 con 0x7f03640a98f0 2024-03-25T18:29:46.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0360001c70 con 0x7f03640a98f0 2024-03-25T18:29:46.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:46.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0360003a60 con 0x7f03640a98f0 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 10 monclient: dump: 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.567 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:46.568 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:46.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f0368d8d640 4 set_mon_vals no callback set 2024-03-25T18:29:46.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.563+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f0360003c80 con 0x7f03640a98f0 2024-03-25T18:29:46.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.564+0000 7f0369d8f640 10 monclient: get_auth_request con 0x7f034806df70 auth_method 0 2024-03-25T18:29:46.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.564+0000 7f035affd640 10 client.4478.objecter ms_handle_connect 0x7f034806df70 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.569+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f0360054860 con 0x7f03640a98f0 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.569+0000 7f035affd640 10 client.4478.objecter ms_dispatch 0x7f0364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.569+0000 7f035affd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.569+0000 7f035affd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.570+0000 7f035affd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:29:46.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.570+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0360050370 con 0x7f03640a98f0 2024-03-25T18:29:46.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.570+0000 7f035affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:46.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.570+0000 7f035affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:46.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.783+0000 7f036c7f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:46.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.783+0000 7f036c7f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:46.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.783+0000 7f036c7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:46.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.783+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0364074cc0 con 0x7f03640a98f0 2024-03-25T18:29:46.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.852+0000 7f035affd640 1 -- 172.21.15.176:0/1004423284 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f03600522a0 con 0x7f03640a98f0 2024-03-25T18:29:46.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.852+0000 7f035affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:46.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.852+0000 7f035affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:46.853 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:46.853 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 10 librados: watch_flush enter 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 10 librados: watch_flush exit 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 >> v1:172.21.15.176:6800/2598213983 conn(0x7f034806df70 legacy=0x7f0348070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 10 monclient: shutdown 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 >> v1:172.21.15.176:6790/0 conn(0x7f03640a98f0 legacy=0x7f03641310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 shutdown_connections 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 >> 172.21.15.176:0/1004423284 conn(0x7f03640b7ea0 msgr2=0x7f0364074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:46.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.854+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 shutdown_connections 2024-03-25T18:29:46.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.855+0000 7f036c7f4640 1 -- 172.21.15.176:0/1004423284 wait complete. 2024-03-25T18:29:46.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:46.855+0000 7f036c7f4640 1 librados: shutdown 2024-03-25T18:29:46.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:53.409 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:55.165 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.458 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:55.192 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.459 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:55.193 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.461 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:55.217 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.465 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:55.222 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.481 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:55.236 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.490 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:55.224 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.508 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:05:55.264 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.520 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:05:55.276 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.674 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:53.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.784+0000 7f4252c43640 10 monclient: get_monmap_and_config 2024-03-25T18:29:53.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.784+0000 7f4252c43640 10 monclient: build_initial_monmap 2024-03-25T18:29:53.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.784+0000 7f4252c43640 10 monclient: monmap: 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:29:53.785365+0000 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:29:53.785365+0000 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:29:53.786 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 1 Processor -- start 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 1 -- start start 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient: init 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-a con 0x7f424c0b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-c con 0x7f424c0b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-b con 0x7f424c14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c0af940 con 0x7f424c0b4d60 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c0b0a70 con 0x7f424c0b1730 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c0b0d50 con 0x7f424c14be90 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:53.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.785+0000 7f4252c43640 10 monclient(hunting): authenticate will time out at 8472.097786s 2024-03-25T18:29:53.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.790+0000 7f42511e0640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f424c14be90 0x7f424c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:34642/0 (socket says 172.21.15.176:34642) 2024-03-25T18:29:53.791 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.790+0000 7f42511e0640 1 -- 172.21.15.176:0/3615658754 learned_addr learned my addr 172.21.15.176:0/3615658754 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090358927 0 0) 0x7f424c0b0d50 con 0x7f424c14be90 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:29:53.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.802+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c003520 con 0x7f424c14be90 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76223167 0 0) 0x7f424c0af940 con 0x7f424c0b4d60 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:29:53.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.816+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c0b0d50 con 0x7f424c0b4d60 2024-03-25T18:29:53.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282320373 0 0) 0x7f424c0b0a70 con 0x7f424c0b1730 2024-03-25T18:29:53.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:29:53.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.820+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c0af940 con 0x7f424c0b1730 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413499595 0 0) 0x7f423c003520 con 0x7f424c14be90 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 >> v1:172.21.15.186:6789/0 conn(0x7f424c0b1730 legacy=0x7f424c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 >> v1:172.21.15.176:6789/0 conn(0x7f424c0b4d60 legacy=0x7f424c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.826+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c0b1030 con 0x7f424c14be90 2024-03-25T18:29:53.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4244001a10 con 0x7f424c14be90 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f4252c43640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f4252c43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f424b7fe640 10 monclient: dump: 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:53.829 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:53.830 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:53.830 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:53.830 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.827+0000 7f4252c43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:53.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4244001b70 con 0x7f424c14be90 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 1 -- 172.21.15.176:0/3615658754 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4244001f80 con 0x7f424c14be90 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f424b7fe640 10 monclient: dump: 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:53.846 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:53.847 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f4252c43640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:53.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.844+0000 7f4252c43640 4 set_mon_vals no callback set 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 10 monclient: shutdown 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 1 -- 172.21.15.176:0/3615658754 >> v1:172.21.15.176:6790/0 conn(0x7f424c14be90 legacy=0x7f424c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 1 -- 172.21.15.176:0/3615658754 shutdown_connections 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 1 -- 172.21.15.176:0/3615658754 >> 172.21.15.176:0/3615658754 conn(0x7f424c0617a0 msgr2=0x7f424c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.845+0000 7f4252c43640 1 -- 172.21.15.176:0/3615658754 shutdown_connections 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.846+0000 7f4252c43640 1 -- 172.21.15.176:0/3615658754 wait complete. 2024-03-25T18:29:53.849 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.846+0000 7f4252c43640 10 monclient: build_initial_monmap 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.846+0000 7f4252c43640 10 monclient: monmap: 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 librados: starting msgr at 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 librados: starting objecter 2024-03-25T18:29:53.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 Processor -- start 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 -- start start 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 librados: setting wanted keys 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 1 librados: calling monclient init 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.847+0000 7f4252c43640 10 monclient: init 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-a con 0x7f424c0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-b con 0x7f424c0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient(hunting): picked mon.noname-c con 0x7f424c14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c140c50 con 0x7f424c0b1730 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c1df7e0 con 0x7f424c0b4d60 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c1dee70 con 0x7f424c14be90 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:53.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.848+0000 7f4252c43640 10 monclient(hunting): authenticate will time out at 8472.160121s 2024-03-25T18:29:53.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.858+0000 7f424bfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f424c0b4d60 0x7f424c1de050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:49158/0 (socket says 172.21.15.176:49158) 2024-03-25T18:29:53.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.859+0000 7f424bfff640 1 -- 172.21.15.176:0/1066540061 learned_addr learned my addr 172.21.15.176:0/1066540061 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:29:53.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.864+0000 7f42497fa640 10 client.?.objecter ms_handle_connect 0x7f424c0b4d60 2024-03-25T18:29:53.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.864+0000 7f42497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:53.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.867+0000 7f42497fa640 10 client.?.objecter ms_handle_connect 0x7f424c14be90 2024-03-25T18:29:53.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.867+0000 7f42497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:53.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.872+0000 7f42497fa640 10 client.?.objecter ms_handle_connect 0x7f424c0b1730 2024-03-25T18:29:53.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.872+0000 7f42497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690103662 0 0) 0x7f424c1dee70 con 0x7f424c14be90 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:29:53.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.881+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4230004eb0 con 0x7f424c14be90 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826571158 0 0) 0x7f424c140c50 con 0x7f424c0b1730 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:29:53.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c1dee70 con 0x7f424c0b1730 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149849191 0 0) 0x7f424c1df7e0 con 0x7f424c0b4d60 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:29:53.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.883+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c140c50 con 0x7f424c0b4d60 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098970855 0 0) 0x7f4230004eb0 con 0x7f424c14be90 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f424c1df7e0 con 0x7f424c14be90 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4244001e80 con 0x7f424c14be90 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 10 monclient(hunting): dump: 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:53.901 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.900+0000 7f42497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:53.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659398901 0 0) 0x7f424c1dee70 con 0x7f424c0b1730 2024-03-25T18:29:53.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4230004eb0 con 0x7f424c0b1730 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4240001970 con 0x7f424c0b1730 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.903+0000 7f42497fa640 10 monclient(hunting): dump: 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:53.905 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:53.906 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:53.906 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 260324948 0 0) 0x7f424c1df7e0 con 0x7f424c14be90 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 >> v1:172.21.15.186:6789/0 conn(0x7f424c0b4d60 legacy=0x7f424c1de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 >> v1:172.21.15.176:6789/0 conn(0x7f424c0b1730 legacy=0x7f424c1df0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 1 monclient: found mon.c 2024-03-25T18:29:53.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c140ec0 con 0x7f424c14be90 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f42497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 10 monclient: _renew_subs 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c1411a0 con 0x7f424c14be90 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 10 monclient: _renew_subs 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:53.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c138340 con 0x7f424c14be90 2024-03-25T18:29:53.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.907+0000 7f4252c43640 1 librados: init done 2024-03-25T18:29:53.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.908+0000 7f4252c43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:53.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.908+0000 7f4252c43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:53.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.908+0000 7f4252c43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:53.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:53.908+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424c0627b0 con 0x7f424c14be90 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4244002020 con 0x7f424c14be90 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4244004fb0 con 0x7f424c14be90 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.199+0000 7f42497fa640 10 monclient: dump: 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:29:54.201 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:29:54.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.200+0000 7f424affd640 4 set_mon_vals no callback set 2024-03-25T18:29:54.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.200+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f424401af80 con 0x7f424c14be90 2024-03-25T18:29:54.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.201+0000 7f42509df640 10 monclient: get_auth_request con 0x7f423006ded0 auth_method 0 2024-03-25T18:29:54.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.201+0000 7f42497fa640 10 client.4484.objecter ms_handle_connect 0x7f423006ded0 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f4244054ae0 con 0x7f424c14be90 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 10 client.4484.objecter ms_dispatch 0x7f424c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4244050420 con 0x7f424c14be90 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:54.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.207+0000 7f42497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:54.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.457+0000 7f4252c43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:54.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.457+0000 7f4252c43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:54.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.457+0000 7f4252c43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:29:54.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.457+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f424c145010 con 0x7f424c14be90 2024-03-25T18:29:54.553 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.552+0000 7f42497fa640 1 -- 172.21.15.176:0/1066540061 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f4244052520 con 0x7f424c14be90 2024-03-25T18:29:54.554 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.552+0000 7f42497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:54.554 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.552+0000 7f42497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:54.554 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:29:54.554 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 10 librados: watch_flush enter 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 10 librados: watch_flush exit 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 10 client.4484.objecter successfully canceled tick 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 >> v1:172.21.15.176:6800/2598213983 conn(0x7f423006ded0 legacy=0x7f4230070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 10 monclient: shutdown 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 >> v1:172.21.15.176:6790/0 conn(0x7f424c14be90 legacy=0x7f424c1de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 shutdown_connections 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.555+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 >> 172.21.15.176:0/1066540061 conn(0x7f424c0617a0 msgr2=0x7f424c13c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.556+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 shutdown_connections 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.556+0000 7f4252c43640 1 -- 172.21.15.176:0/1066540061 wait complete. 2024-03-25T18:29:54.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:29:54.556+0000 7f4252c43640 1 librados: shutdown 2024-03-25T18:29:54.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:01.376 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:01.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.489+0000 7f1b9803e640 10 monclient: get_monmap_and_config 2024-03-25T18:30:01.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.489+0000 7f1b9803e640 10 monclient: build_initial_monmap 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.489+0000 7f1b9803e640 10 monclient: monmap: 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:01.490080+0000 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:01.490080+0000 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:01.491 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:01.492 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.489+0000 7f1b9803e640 1 Processor -- start 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 1 -- start start 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient: init 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b900a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b9013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b901430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90142520 con 0x7f1b9013f710 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900b28a0 con 0x7f1b900a98f0 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900b2a30 con 0x7f1b901430c0 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:01.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.490+0000 7f1b9803e640 10 monclient(hunting): authenticate will time out at 8479.802816s 2024-03-25T18:30:01.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.505+0000 7f1b95dda640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1b900a98f0 0x7f1b900a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:43808/0 (socket says 172.21.15.176:43808) 2024-03-25T18:30:01.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.505+0000 7f1b95dda640 1 -- 172.21.15.176:0/1503312044 learned_addr learned my addr 172.21.15.176:0/1503312044 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577322588 0 0) 0x7f1b900b2a30 con 0x7f1b901430c0 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 10 monclient(hunting): _negotiate 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:30:01.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.513+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c003520 con 0x7f1b901430c0 2024-03-25T18:30:01.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715252764 0 0) 0x7f1b900b28a0 con 0x7f1b900a98f0 2024-03-25T18:30:01.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 10 monclient(hunting): _negotiate 2024-03-25T18:30:01.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:01.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:01.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:30:01.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.518+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b900b2a30 con 0x7f1b900a98f0 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490046817 0 0) 0x7f1b7c003520 con 0x7f1b901430c0 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 >> v1:172.21.15.186:6789/0 conn(0x7f1b900a98f0 legacy=0x7f1b900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 >> v1:172.21.15.176:6789/0 conn(0x7f1b9013f710 legacy=0x7f1b901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: _finish_hunting 0 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 monclient: found mon.noname-b 2024-03-25T18:30:01.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900b44c0 con 0x7f1b901430c0 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: _finish_auth 0 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: _check_auth_tickets 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1b88001a10 con 0x7f1b901430c0 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b9803e640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b9803e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b94dd8640 10 monclient: dump: 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.524 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.522+0000 7f1b9803e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b88001b70 con 0x7f1b901430c0 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 1 -- 172.21.15.176:0/1503312044 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1b88001f80 con 0x7f1b901430c0 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b94dd8640 10 monclient: dump: 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:01.529 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:01.530 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:01.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 4 set_mon_vals no callback set 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 10 monclient: shutdown 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 1 -- 172.21.15.176:0/1503312044 >> v1:172.21.15.176:6790/0 conn(0x7f1b901430c0 legacy=0x7f1b9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 1 -- 172.21.15.176:0/1503312044 shutdown_connections 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.528+0000 7f1b9803e640 1 -- 172.21.15.176:0/1503312044 >> 172.21.15.176:0/1503312044 conn(0x7f1b900b7ea0 msgr2=0x7f1b900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.529+0000 7f1b9803e640 1 -- 172.21.15.176:0/1503312044 shutdown_connections 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.529+0000 7f1b9803e640 1 -- 172.21.15.176:0/1503312044 wait complete. 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.529+0000 7f1b9803e640 10 monclient: build_initial_monmap 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.529+0000 7f1b9803e640 10 monclient: monmap: 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:01.533 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 librados: starting msgr at 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 librados: starting objecter 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 Processor -- start 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 -- start start 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 librados: setting wanted keys 2024-03-25T18:30:01.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 1 librados: calling monclient init 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.530+0000 7f1b9803e640 10 monclient: init 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b900a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b9013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b901430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b901dc070 con 0x7f1b901430c0 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90136bd0 con 0x7f1b900a98f0 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90136eb0 con 0x7f1b9013f710 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:01.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.531+0000 7f1b9803e640 10 monclient(hunting): authenticate will time out at 8479.843070s 2024-03-25T18:30:01.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.536+0000 7f1b955d9640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f1b9013f710 0x7f1b901db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:38402/0 (socket says 172.21.15.176:38402) 2024-03-25T18:30:01.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.536+0000 7f1b955d9640 1 -- 172.21.15.176:0/1311443885 learned_addr learned my addr 172.21.15.176:0/1311443885 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:01.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.537+0000 7f1b867fc640 10 client.?.objecter ms_handle_connect 0x7f1b9013f710 2024-03-25T18:30:01.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.537+0000 7f1b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890938941 0 0) 0x7f1b90136eb0 con 0x7f1b9013f710 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:30:01.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b74004eb0 con 0x7f1b9013f710 2024-03-25T18:30:01.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 client.?.objecter ms_handle_connect 0x7f1b900a98f0 2024-03-25T18:30:01.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.544+0000 7f1b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:01.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.554+0000 7f1b867fc640 10 client.?.objecter ms_handle_connect 0x7f1b901430c0 2024-03-25T18:30:01.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.555+0000 7f1b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704838881 0 0) 0x7f1b90136bd0 con 0x7f1b900a98f0 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:30:01.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.557+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b90136eb0 con 0x7f1b900a98f0 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055525513 0 0) 0x7f1b901dc070 con 0x7f1b901430c0 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:30:01.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.571+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b90136bd0 con 0x7f1b901430c0 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2582985142 0 0) 0x7f1b74004eb0 con 0x7f1b9013f710 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b901dc070 con 0x7f1b9013f710 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1b80001a30 con 0x7f1b9013f710 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 10 monclient(hunting): dump: 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:01.574 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:01.575 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:01.575 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.572+0000 7f1b867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154708828 0 0) 0x7f1b90136eb0 con 0x7f1b900a98f0 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b74004eb0 con 0x7f1b900a98f0 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1b8c0018b0 con 0x7f1b900a98f0 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.580+0000 7f1b867fc640 10 monclient(hunting): dump: 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.581 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:01.582 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.582 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:01.582 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:01.582 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:01.582 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3354335314 0 0) 0x7f1b901dc070 con 0x7f1b9013f710 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 >> v1:172.21.15.186:6789/0 conn(0x7f1b900a98f0 legacy=0x7f1b90137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 >> v1:172.21.15.176:6789/0 conn(0x7f1b901430c0 legacy=0x7f1b901db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 1 monclient: found mon.c 2024-03-25T18:30:01.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90137190 con 0x7f1b9013f710 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 10 monclient: _renew_subs 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b90137470 con 0x7f1b9013f710 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 10 monclient: _renew_subs 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b901355e0 con 0x7f1b9013f710 2024-03-25T18:30:01.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.582+0000 7f1b9803e640 1 librados: init done 2024-03-25T18:30:01.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.583+0000 7f1b9803e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:01.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.583+0000 7f1b9803e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:01.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.583+0000 7f1b9803e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:01.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.583+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b90135830 con 0x7f1b9013f710 2024-03-25T18:30:01.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b80001c70 con 0x7f1b9013f710 2024-03-25T18:30:01.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f1b80003a60 con 0x7f1b9013f710 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b867fc640 10 monclient: dump: 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:01.873 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.871+0000 7f1b87fff640 4 set_mon_vals no callback set 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.872+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f1b80003c80 con 0x7f1b9013f710 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.872+0000 7f1b95dda640 10 monclient: get_auth_request con 0x7f1b7406df70 auth_method 0 2024-03-25T18:30:01.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.872+0000 7f1b867fc640 10 client.4490.objecter ms_handle_connect 0x7f1b7406df70 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f1b800548a0 con 0x7f1b9013f710 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 10 client.4490.objecter ms_dispatch 0x7f1b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b800503b0 con 0x7f1b9013f710 2024-03-25T18:30:01.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:01.881 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:01.879+0000 7f1b867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:02.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.099+0000 7f1b9803e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:02.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.099+0000 7f1b9803e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:02.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.099+0000 7f1b9803e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:02.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.099+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b900627b0 con 0x7f1b9013f710 2024-03-25T18:30:02.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.128+0000 7f1b867fc640 1 -- 172.21.15.176:0/1311443885 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f1b800522e0 con 0x7f1b9013f710 2024-03-25T18:30:02.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.128+0000 7f1b867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:02.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.128+0000 7f1b867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:02.130 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:02.130 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:02.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 10 librados: watch_flush enter 2024-03-25T18:30:02.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 10 librados: watch_flush exit 2024-03-25T18:30:02.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 >> v1:172.21.15.176:6800/2598213983 conn(0x7f1b7406df70 legacy=0x7f1b74070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 10 monclient: shutdown 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 >> v1:172.21.15.176:6790/0 conn(0x7f1b9013f710 legacy=0x7f1b901db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 shutdown_connections 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 >> 172.21.15.176:0/1311443885 conn(0x7f1b900b7ea0 msgr2=0x7f1b9013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 shutdown_connections 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 -- 172.21.15.176:0/1311443885 wait complete. 2024-03-25T18:30:02.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:02.131+0000 7f1b9803e640 1 librados: shutdown 2024-03-25T18:30:02.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:08.952 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:09.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.069+0000 7f6b486a8640 10 monclient: get_monmap_and_config 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.069+0000 7f6b486a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.069+0000 7f6b486a8640 10 monclient: monmap: 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:09.070583+0000 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:09.070583+0000 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:09.071 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.070+0000 7f6b486a8640 1 Processor -- start 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.070+0000 7f6b486a8640 1 -- start start 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.070+0000 7f6b486a8640 10 monclient: init 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6b400a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6b401430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40142520 con 0x7f6b401430c0 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400b28a0 con 0x7f6b4013f710 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400b2a30 con 0x7f6b400a98f0 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:09.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.071+0000 7f6b486a8640 10 monclient(hunting): authenticate will time out at 8487.383216s 2024-03-25T18:30:09.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.081+0000 7f6b46c45640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f6b401430c0 0x7f6b4014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:41628/0 (socket says 172.21.15.176:41628) 2024-03-25T18:30:09.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.081+0000 7f6b46c45640 1 -- 172.21.15.176:0/2431388906 learned_addr learned my addr 172.21.15.176:0/2431388906 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:09.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285759253 0 0) 0x7f6b40142520 con 0x7f6b401430c0 2024-03-25T18:30:09.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 10 monclient(hunting): _negotiate 2024-03-25T18:30:09.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:09.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:09.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:30:09.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.086+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c003520 con 0x7f6b401430c0 2024-03-25T18:30:09.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279311860 0 0) 0x7f6b400b28a0 con 0x7f6b4013f710 2024-03-25T18:30:09.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 10 monclient(hunting): _negotiate 2024-03-25T18:30:09.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:09.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:09.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:30:09.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.095+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b40142520 con 0x7f6b4013f710 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591479998 0 0) 0x7f6b2c003520 con 0x7f6b401430c0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 >> v1:172.21.15.176:6790/0 conn(0x7f6b400a98f0 legacy=0x7f6b400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 >> v1:172.21.15.186:6789/0 conn(0x7f6b4013f710 legacy=0x7f6b401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: _finish_hunting 0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 monclient: found mon.noname-a 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b400b44c0 con 0x7f6b401430c0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: _finish_auth 0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: _check_auth_tickets 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6b38001a10 con 0x7f6b401430c0 2024-03-25T18:30:09.098 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b45442640 10 monclient: dump: 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b486a8640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:30:09.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.096+0000 7f6b486a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b38001b70 con 0x7f6b401430c0 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 1 -- 172.21.15.176:0/2431388906 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6b38001f80 con 0x7f6b401430c0 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b45442640 10 monclient: dump: 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.107 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b486a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:09.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b486a8640 4 set_mon_vals no callback set 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b486a8640 10 monclient: shutdown 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b486a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.106+0000 7f6b486a8640 1 -- 172.21.15.176:0/2431388906 >> v1:172.21.15.176:6789/0 conn(0x7f6b401430c0 legacy=0x7f6b4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 1 -- 172.21.15.176:0/2431388906 shutdown_connections 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 1 -- 172.21.15.176:0/2431388906 >> 172.21.15.176:0/2431388906 conn(0x7f6b400b7ea0 msgr2=0x7f6b400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 1 -- 172.21.15.176:0/2431388906 shutdown_connections 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 1 -- 172.21.15.176:0/2431388906 wait complete. 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:09.110 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.107+0000 7f6b486a8640 10 monclient: monmap: 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 librados: starting msgr at 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 librados: starting objecter 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 Processor -- start 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 -- start start 2024-03-25T18:30:09.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 librados: setting wanted keys 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 librados: calling monclient init 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient: init 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6b400a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6b401430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40136c50 con 0x7f6b400a98f0 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40136ec0 con 0x7f6b401430c0 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b401317f0 con 0x7f6b4013f710 2024-03-25T18:30:09.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:09.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.108+0000 7f6b486a8640 10 monclient(hunting): authenticate will time out at 8487.420811s 2024-03-25T18:30:09.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.114+0000 7f6b46c45640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6b401430c0 0x7f6b401db380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:52356/0 (socket says 172.21.15.176:52356) 2024-03-25T18:30:09.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.114+0000 7f6b46c45640 1 -- 172.21.15.176:0/3854401858 learned_addr learned my addr 172.21.15.176:0/3854401858 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:09.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.115+0000 7f6b36ffd640 10 client.?.objecter ms_handle_connect 0x7f6b401430c0 2024-03-25T18:30:09.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.115+0000 7f6b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:09.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.118+0000 7f6b36ffd640 10 client.?.objecter ms_handle_connect 0x7f6b400a98f0 2024-03-25T18:30:09.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.118+0000 7f6b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:09.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.123+0000 7f6b36ffd640 10 client.?.objecter ms_handle_connect 0x7f6b4013f710 2024-03-25T18:30:09.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.123+0000 7f6b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089765449 0 0) 0x7f6b40136c50 con 0x7f6b400a98f0 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:30:09.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.124+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b24004eb0 con 0x7f6b400a98f0 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.126+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152298735 0 0) 0x7f6b40136ec0 con 0x7f6b401430c0 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.126+0000 7f6b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.126+0000 7f6b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.126+0000 7f6b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.126+0000 7f6b36ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:30:09.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.127+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b40136c50 con 0x7f6b401430c0 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461750360 0 0) 0x7f6b24004eb0 con 0x7f6b400a98f0 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b40136ec0 con 0x7f6b400a98f0 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6b3c001930 con 0x7f6b400a98f0 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:09.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.133+0000 7f6b36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:09.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283248561 0 0) 0x7f6b401317f0 con 0x7f6b4013f710 2024-03-25T18:30:09.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:09.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:09.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:09.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:30:09.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.136+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b24004eb0 con 0x7f6b4013f710 2024-03-25T18:30:09.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744546307 0 0) 0x7f6b40136ec0 con 0x7f6b400a98f0 2024-03-25T18:30:09.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 >> v1:172.21.15.176:6790/0 conn(0x7f6b4013f710 legacy=0x7f6b401dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 >> v1:172.21.15.186:6789/0 conn(0x7f6b401430c0 legacy=0x7f6b401db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 1 monclient: found mon.a 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40131980 con 0x7f6b400a98f0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 10 monclient: _renew_subs 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b40135070 con 0x7f6b400a98f0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 10 monclient: _renew_subs 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b400b7870 con 0x7f6b400a98f0 2024-03-25T18:30:09.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.145+0000 7f6b486a8640 1 librados: init done 2024-03-25T18:30:09.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.146+0000 7f6b486a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:09.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.146+0000 7f6b486a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:09.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.146+0000 7f6b486a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.146+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400627b0 con 0x7f6b400a98f0 2024-03-25T18:30:09.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b3c003610 con 0x7f6b400a98f0 2024-03-25T18:30:09.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:09.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6b3c0039a0 con 0x7f6b400a98f0 2024-03-25T18:30:09.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:09.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b36ffd640 10 monclient: dump: 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:09.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.448+0000 7f6b44c41640 4 set_mon_vals no callback set 2024-03-25T18:30:09.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.449+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f6b3c003bc0 con 0x7f6b400a98f0 2024-03-25T18:30:09.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.449+0000 7f6b45c43640 10 monclient: get_auth_request con 0x7f6b2406de00 auth_method 0 2024-03-25T18:30:09.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.449+0000 7f6b36ffd640 10 client.4473.objecter ms_handle_connect 0x7f6b2406de00 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f6b3c053650 con 0x7f6b400a98f0 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 10 client.4473.objecter ms_dispatch 0x7f6b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:30:09.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b3c04f160 con 0x7f6b400a98f0 2024-03-25T18:30:09.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:09.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.458+0000 7f6b36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:09.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.687+0000 7f6b486a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:09.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.687+0000 7f6b486a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:09.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.687+0000 7f6b486a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:09.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.687+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b400b7ac0 con 0x7f6b400a98f0 2024-03-25T18:30:09.722 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.720+0000 7f6b36ffd640 1 -- 172.21.15.176:0/3854401858 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f6b3c051090 con 0x7f6b400a98f0 2024-03-25T18:30:09.722 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.720+0000 7f6b36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:09.722 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.720+0000 7f6b36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:09.722 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:09.722 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:09.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.722+0000 7f6b486a8640 10 librados: watch_flush enter 2024-03-25T18:30:09.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 10 librados: watch_flush exit 2024-03-25T18:30:09.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:30:09.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 >> v1:172.21.15.176:6800/2598213983 conn(0x7f6b2406de00 legacy=0x7f6b240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 10 monclient: shutdown 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 >> v1:172.21.15.176:6789/0 conn(0x7f6b400a98f0 legacy=0x7f6b401310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 shutdown_connections 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 >> 172.21.15.176:0/3854401858 conn(0x7f6b400b7ea0 msgr2=0x7f6b4013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 shutdown_connections 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 -- 172.21.15.176:0/3854401858 wait complete. 2024-03-25T18:30:09.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:09.723+0000 7f6b486a8640 1 librados: shutdown 2024-03-25T18:30:09.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:13.509 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:15.264 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.509 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:15.243 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.511 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:15.245 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.514 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:15.270 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.548 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:15.282 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.574 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:15.330 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.580 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:15.336 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.580 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:15.336 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.599 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:15.355 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.546 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:16.661 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.660+0000 7efd2b7a8640 10 monclient: get_monmap_and_config 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.660+0000 7efd2b7a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.660+0000 7efd2b7a8640 10 monclient: monmap: 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:16.661437+0000 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:16.661437+0000 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:16.662 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.661+0000 7efd2b7a8640 1 Processor -- start 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.661+0000 7efd2b7a8640 1 -- start start 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.661+0000 7efd2b7a8640 10 monclient: init 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-c con 0x7efd240a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-a con 0x7efd2413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-b con 0x7efd241430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd24142520 con 0x7efd2413f710 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240b28a0 con 0x7efd240a98f0 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240b2a30 con 0x7efd241430c0 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:16.664 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.662+0000 7efd2b7a8640 10 monclient(hunting): authenticate will time out at 8494.974099s 2024-03-25T18:30:16.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.670+0000 7efd29d45640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7efd241430c0 0x7efd2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53104/0 (socket says 172.21.15.176:53104) 2024-03-25T18:30:16.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.670+0000 7efd29d45640 1 -- 172.21.15.176:0/4287559146 learned_addr learned my addr 172.21.15.176:0/4287559146 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:16.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013184720 0 0) 0x7efd240b2a30 con 0x7efd241430c0 2024-03-25T18:30:16.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:16.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:16.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:16.682 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:30:16.683 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.681+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10003520 con 0x7efd241430c0 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808077153 0 0) 0x7efd240b28a0 con 0x7efd240a98f0 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:30:16.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.685+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd240b2a30 con 0x7efd240a98f0 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39447544 0 0) 0x7efd240b2a30 con 0x7efd240a98f0 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 >> v1:172.21.15.176:6790/0 conn(0x7efd241430c0 legacy=0x7efd2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 >> v1:172.21.15.176:6789/0 conn(0x7efd2413f710 legacy=0x7efd241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 monclient: found mon.noname-c 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd240b44c0 con 0x7efd240a98f0 2024-03-25T18:30:16.692 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd20001670 con 0x7efd240a98f0 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.690+0000 7efd1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.691+0000 7efd1bfff640 10 monclient: dump: 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:16.693 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:16.694 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:16.694 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:16.694 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.694 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.691+0000 7efd2b7a8640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:30:16.694 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.691+0000 7efd2b7a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:16.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd200017d0 con 0x7efd240a98f0 2024-03-25T18:30:16.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:16.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 1 -- 172.21.15.176:0/4287559146 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd20001b60 con 0x7efd240a98f0 2024-03-25T18:30:16.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd2b7a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd2b7a8640 4 set_mon_vals no callback set 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.694+0000 7efd1bfff640 10 monclient: dump: 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:16.696 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 10 monclient: shutdown 2024-03-25T18:30:16.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:16.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4287559146 >> v1:172.21.15.186:6789/0 conn(0x7efd240a98f0 legacy=0x7efd240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:16.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4287559146 shutdown_connections 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4287559146 >> 172.21.15.176:0/4287559146 conn(0x7efd240b7ea0 msgr2=0x7efd240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4287559146 shutdown_connections 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.695+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4287559146 wait complete. 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.696+0000 7efd2b7a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.696+0000 7efd2b7a8640 10 monclient: monmap: 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 librados: starting msgr at 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 librados: starting objecter 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 Processor -- start 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 -- start start 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 librados: setting wanted keys 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 librados: calling monclient init 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient: init 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-c con 0x7efd240a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-b con 0x7efd2413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient(hunting): picked mon.noname-a con 0x7efd241430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd241390e0 con 0x7efd241430c0 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd241349e0 con 0x7efd2413f710 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd24139350 con 0x7efd240a98f0 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:16.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.697+0000 7efd2b7a8640 10 monclient(hunting): authenticate will time out at 8495.009820s 2024-03-25T18:30:16.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.703+0000 7efd29544640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7efd240a98f0 0x7efd241d72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53130/0 (socket says 172.21.15.176:53130) 2024-03-25T18:30:16.704 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.704+0000 7efd29544640 1 -- 172.21.15.176:0/4029382945 learned_addr learned my addr 172.21.15.176:0/4029382945 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:16.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.707+0000 7efd19ffb640 10 client.?.objecter ms_handle_connect 0x7efd2413f710 2024-03-25T18:30:16.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.707+0000 7efd19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:16.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.708+0000 7efd19ffb640 10 client.?.objecter ms_handle_connect 0x7efd240a98f0 2024-03-25T18:30:16.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.708+0000 7efd19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:16.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.710+0000 7efd19ffb640 10 client.?.objecter ms_handle_connect 0x7efd241430c0 2024-03-25T18:30:16.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.710+0000 7efd19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:16.719 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281427913 0 0) 0x7efd241390e0 con 0x7efd241430c0 2024-03-25T18:30:16.719 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:16.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:16.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:16.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:30:16.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.718+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd08004eb0 con 0x7efd241430c0 2024-03-25T18:30:16.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782712094 0 0) 0x7efd241349e0 con 0x7efd2413f710 2024-03-25T18:30:16.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:16.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:16.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:16.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:30:16.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.790+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd241390e0 con 0x7efd2413f710 2024-03-25T18:30:16.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551364245 0 0) 0x7efd08004eb0 con 0x7efd241430c0 2024-03-25T18:30:16.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd241349e0 con 0x7efd241430c0 2024-03-25T18:30:16.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd1c001af0 con 0x7efd241430c0 2024-03-25T18:30:16.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:16.794 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.791+0000 7efd19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786241148 0 0) 0x7efd24139350 con 0x7efd240a98f0 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:30:16.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.797+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd08004eb0 con 0x7efd240a98f0 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14355437 0 0) 0x7efd08004eb0 con 0x7efd240a98f0 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd24139350 con 0x7efd240a98f0 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd20001ae0 con 0x7efd240a98f0 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.811+0000 7efd19ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:16.812 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:16.813 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:16.814 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 297057137 0 0) 0x7efd241349e0 con 0x7efd241430c0 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 >> v1:172.21.15.176:6790/0 conn(0x7efd240a98f0 legacy=0x7efd241d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 >> v1:172.21.15.186:6789/0 conn(0x7efd2413f710 legacy=0x7efd24133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 1 monclient: found mon.a 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24139630 con 0x7efd241430c0 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 10 monclient: _renew_subs 2024-03-25T18:30:16.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd24139910 con 0x7efd241430c0 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 10 monclient: _renew_subs 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd24137800 con 0x7efd241430c0 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.812+0000 7efd2b7a8640 1 librados: init done 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.813+0000 7efd2b7a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.813+0000 7efd2b7a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.813+0000 7efd2b7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:16.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:16.813+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd24138410 con 0x7efd241430c0 2024-03-25T18:30:17.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd1c001f20 con 0x7efd241430c0 2024-03-25T18:30:17.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:17.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd1c004d00 con 0x7efd241430c0 2024-03-25T18:30:17.099 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd19ffb640 10 monclient: dump: 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:17.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.098+0000 7efd1b7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:17.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.099+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7efd1c004f20 con 0x7efd241430c0 2024-03-25T18:30:17.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.099+0000 7efd29544640 10 monclient: get_auth_request con 0x7efd0806dfb0 auth_method 0 2024-03-25T18:30:17.101 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.100+0000 7efd19ffb640 10 client.4479.objecter ms_handle_connect 0x7efd0806dfb0 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7efd1c054970 con 0x7efd241430c0 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 10 client.4479.objecter ms_dispatch 0x7efd24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd1c050480 con 0x7efd241430c0 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:17.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.104+0000 7efd19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:17.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.361+0000 7efd2b7a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:17.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.361+0000 7efd2b7a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:17.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.361+0000 7efd2b7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:30:17.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.361+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd24074cc0 con 0x7efd241430c0 2024-03-25T18:30:17.385 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.384+0000 7efd19ffb640 1 -- 172.21.15.176:0/4029382945 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7efd1c0523b0 con 0x7efd241430c0 2024-03-25T18:30:17.385 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.384+0000 7efd19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:17.385 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.384+0000 7efd19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:17.385 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:17.385 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:17.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.386+0000 7efd2b7a8640 10 librados: watch_flush enter 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.386+0000 7efd2b7a8640 10 librados: watch_flush exit 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.386+0000 7efd2b7a8640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.386+0000 7efd2b7a8640 10 client.4479.objecter successfully canceled tick 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 >> v1:172.21.15.176:6800/2598213983 conn(0x7efd0806dfb0 legacy=0x7efd08070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 10 monclient: shutdown 2024-03-25T18:30:17.388 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 >> v1:172.21.15.176:6789/0 conn(0x7efd241430c0 legacy=0x7efd241342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 shutdown_connections 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 >> 172.21.15.176:0/4029382945 conn(0x7efd240b7ea0 msgr2=0x7efd24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 shutdown_connections 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 -- 172.21.15.176:0/4029382945 wait complete. 2024-03-25T18:30:17.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:17.387+0000 7efd2b7a8640 1 librados: shutdown 2024-03-25T18:30:17.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:24.206 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:24.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.306+0000 7fe12defb640 10 monclient: get_monmap_and_config 2024-03-25T18:30:24.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.306+0000 7fe12defb640 10 monclient: build_initial_monmap 2024-03-25T18:30:24.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.306+0000 7fe12defb640 10 monclient: monmap: 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:24.307194+0000 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:24.307194+0000 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:24.308 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 1 Processor -- start 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 1 -- start start 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient: init 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-c con 0x7fe1280a96e0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:24.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-b con 0x7fe1280b1b10 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-a con 0x7fe128146060 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe128140040 con 0x7fe128146060 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1280b1fa0 con 0x7fe1280a96e0 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1280b24c0 con 0x7fe1280b1b10 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:24.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.307+0000 7fe12defb640 10 monclient(hunting): authenticate will time out at 8502.619763s 2024-03-25T18:30:24.323 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.322+0000 7fe127fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe128146060 0x7fe12814dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:56828/0 (socket says 172.21.15.176:56828) 2024-03-25T18:30:24.323 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.322+0000 7fe127fff640 1 -- 172.21.15.176:0/4113463976 learned_addr learned my addr 172.21.15.176:0/4113463976 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:24.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484250452 0 0) 0x7fe128140040 con 0x7fe128146060 2024-03-25T18:30:24.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:24.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:24.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:24.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:30:24.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.331+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe114003520 con 0x7fe128146060 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3160857321 0 0) 0x7fe114003520 con 0x7fe128146060 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 >> v1:172.21.15.176:6790/0 conn(0x7fe1280b1b10 legacy=0x7fe128145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 >> v1:172.21.15.186:6789/0 conn(0x7fe1280a96e0 legacy=0x7fe1280b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:24.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.348+0000 7fe126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1280b15b0 con 0x7fe128146060 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe120001a10 con 0x7fe128146060 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe12defb640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe12defb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe126ffd640 10 monclient: dump: 2024-03-25T18:30:24.351 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.349+0000 7fe12defb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe120001be0 con 0x7fe128146060 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 1 -- 172.21.15.176:0/4113463976 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe120002030 con 0x7fe128146060 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe12defb640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe12defb640 4 set_mon_vals no callback set 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.385+0000 7fe126ffd640 10 monclient: dump: 2024-03-25T18:30:24.387 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:24.388 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 10 monclient: shutdown 2024-03-25T18:30:24.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 1 -- 172.21.15.176:0/4113463976 >> v1:172.21.15.176:6789/0 conn(0x7fe128146060 legacy=0x7fe12814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 1 -- 172.21.15.176:0/4113463976 shutdown_connections 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 1 -- 172.21.15.176:0/4113463976 >> 172.21.15.176:0/4113463976 conn(0x7fe1280b7ea0 msgr2=0x7fe1280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 1 -- 172.21.15.176:0/4113463976 shutdown_connections 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.386+0000 7fe12defb640 1 -- 172.21.15.176:0/4113463976 wait complete. 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.387+0000 7fe12defb640 10 monclient: build_initial_monmap 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.387+0000 7fe12defb640 10 monclient: monmap: 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:24.391 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 librados: starting msgr at 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 librados: starting objecter 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 Processor -- start 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 -- start start 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 librados: setting wanted keys 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 1 librados: calling monclient init 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 10 monclient: init 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.388+0000 7fe12defb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:24.392 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-b con 0x7fe1280a96e0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-c con 0x7fe128146060 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 10 monclient(hunting): picked mon.noname-a con 0x7fe1280ab9e0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe128143090 con 0x7fe1280ab9e0 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12813e560 con 0x7fe1280a96e0 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12813e6f0 con 0x7fe128146060 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:24.393 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.389+0000 7fe12defb640 10 monclient(hunting): authenticate will time out at 8502.701000s 2024-03-25T18:30:24.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.395+0000 7fe11ffff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fe128146060 0x7fe1281d8280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:57860/0 (socket says 172.21.15.176:57860) 2024-03-25T18:30:24.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.395+0000 7fe11ffff640 1 -- 172.21.15.176:0/1409450345 learned_addr learned my addr 172.21.15.176:0/1409450345 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:24.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.396+0000 7fe124ff9640 10 client.?.objecter ms_handle_connect 0x7fe128146060 2024-03-25T18:30:24.397 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.396+0000 7fe124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:24.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.406+0000 7fe124ff9640 10 client.?.objecter ms_handle_connect 0x7fe1280ab9e0 2024-03-25T18:30:24.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.406+0000 7fe124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:24.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.408+0000 7fe124ff9640 10 client.?.objecter ms_handle_connect 0x7fe1280a96e0 2024-03-25T18:30:24.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.408+0000 7fe124ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096055431 0 0) 0x7fe12813e6f0 con 0x7fe128146060 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:30:24.410 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.409+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108004eb0 con 0x7fe128146060 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422048930 0 0) 0x7fe12813e560 con 0x7fe1280a96e0 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:30:24.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.416+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe12813e6f0 con 0x7fe1280a96e0 2024-03-25T18:30:24.422 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2782611472 0 0) 0x7fe108004eb0 con 0x7fe128146060 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe12813e560 con 0x7fe128146060 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe1100019b0 con 0x7fe128146060 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:24.423 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:24.424 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:24.424 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.424 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.421+0000 7fe124ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:24.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031710528 0 0) 0x7fe128143090 con 0x7fe1280ab9e0 2024-03-25T18:30:24.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:24.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:24.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:24.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:30:24.437 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.435+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108004eb0 con 0x7fe1280ab9e0 2024-03-25T18:30:24.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2259564514 0 0) 0x7fe12813e560 con 0x7fe128146060 2024-03-25T18:30:24.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 >> v1:172.21.15.186:6789/0 conn(0x7fe1280a96e0 legacy=0x7fe1281d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:24.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 >> v1:172.21.15.176:6789/0 conn(0x7fe1280ab9e0 legacy=0x7fe1281e24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:24.449 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 1 monclient: found mon.c 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12813e880 con 0x7fe128146060 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe124ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 10 monclient: _renew_subs 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:24.450 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1281429a0 con 0x7fe128146060 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 10 monclient: _renew_subs 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1281e3bd0 con 0x7fe128146060 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.448+0000 7fe12defb640 1 librados: init done 2024-03-25T18:30:24.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.449+0000 7fe12defb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:24.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.449+0000 7fe12defb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:24.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.449+0000 7fe12defb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:24.452 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.449+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1280a9ac0 con 0x7fe128146060 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1100035f0 con 0x7fe128146060 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe110003940 con 0x7fe128146060 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe124ff9640 10 monclient: dump: 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:24.745 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:24.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.743+0000 7fe1267fc640 4 set_mon_vals no callback set 2024-03-25T18:30:24.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.744+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fe110003b60 con 0x7fe128146060 2024-03-25T18:30:24.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.745+0000 7fe1277fe640 10 monclient: get_auth_request con 0x7fe10806de00 auth_method 0 2024-03-25T18:30:24.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.745+0000 7fe124ff9640 10 client.4505.objecter ms_handle_connect 0x7fe10806de00 2024-03-25T18:30:24.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fe110054790 con 0x7fe128146060 2024-03-25T18:30:24.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 10 client.4505.objecter ms_dispatch 0x7fe128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:24.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:24.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:24.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T18:30:24.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1100502a0 con 0x7fe128146060 2024-03-25T18:30:24.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:24.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.750+0000 7fe124ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:24.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.998+0000 7fe12defb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:24.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.998+0000 7fe12defb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:24.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.998+0000 7fe12defb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:25.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:24.998+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe128142e80 con 0x7fe128146060 2024-03-25T18:30:25.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.039+0000 7fe124ff9640 1 -- 172.21.15.176:0/1409450345 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fe1100521d0 con 0x7fe128146060 2024-03-25T18:30:25.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.039+0000 7fe124ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:25.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.039+0000 7fe124ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:25.041 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:25.041 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:25.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 10 librados: watch_flush enter 2024-03-25T18:30:25.043 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 10 librados: watch_flush exit 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 10 client.4505.objecter successfully canceled tick 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 >> v1:172.21.15.176:6800/2598213983 conn(0x7fe10806de00 legacy=0x7fe1080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 10 monclient: shutdown 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 >> v1:172.21.15.176:6790/0 conn(0x7fe128146060 legacy=0x7fe1281d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 shutdown_connections 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 >> 172.21.15.176:0/1409450345 conn(0x7fe1280b7ea0 msgr2=0x7fe1280b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 shutdown_connections 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 -- 172.21.15.176:0/1409450345 wait complete. 2024-03-25T18:30:25.044 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:25.042+0000 7fe12defb640 1 librados: shutdown 2024-03-25T18:30:25.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:31.863 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:31.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.974+0000 7ff286182640 10 monclient: get_monmap_and_config 2024-03-25T18:30:31.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.974+0000 7ff286182640 10 monclient: build_initial_monmap 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.974+0000 7ff286182640 10 monclient: monmap: 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:31.976584+0000 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:31.976584+0000 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:31.977 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:31.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.975+0000 7ff286182640 1 Processor -- start 2024-03-25T18:30:31.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.975+0000 7ff286182640 1 -- start start 2024-03-25T18:30:31.978 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.975+0000 7ff286182640 10 monclient: init 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient(hunting): picked mon.noname-c con 0x7ff2800a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient(hunting): picked mon.noname-a con 0x7ff28013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient(hunting): picked mon.noname-b con 0x7ff2801430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280142520 con 0x7ff28013f710 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800b28a0 con 0x7ff2800a98f0 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800b2a30 con 0x7ff2801430c0 2024-03-25T18:30:31.979 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:31.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.976+0000 7ff286182640 10 monclient(hunting): authenticate will time out at 8510.289604s 2024-03-25T18:30:31.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.994+0000 7ff27ffff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7ff2801430c0 0x7ff28014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:35812/0 (socket says 172.21.15.176:35812) 2024-03-25T18:30:31.996 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:31.994+0000 7ff27ffff640 1 -- 172.21.15.176:0/1804058891 learned_addr learned my addr 172.21.15.176:0/1804058891 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:32.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576504886 0 0) 0x7ff2800b2a30 con 0x7ff2801430c0 2024-03-25T18:30:32.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:32.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:32.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:32.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:30:32.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.008+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c003520 con 0x7ff2801430c0 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510568008 0 0) 0x7ff280142520 con 0x7ff28013f710 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:30:32.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.015+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2800b2a30 con 0x7ff28013f710 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.020+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66878041 0 0) 0x7ff2800b28a0 con 0x7ff2800a98f0 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.020+0000 7ff27effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.020+0000 7ff27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.020+0000 7ff27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.020+0000 7ff27effd640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:30:32.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.021+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280142520 con 0x7ff2800a98f0 2024-03-25T18:30:32.024 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267623855 0 0) 0x7ff26c003520 con 0x7ff2801430c0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 >> v1:172.21.15.186:6789/0 conn(0x7ff2800a98f0 legacy=0x7ff2800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 >> v1:172.21.15.176:6789/0 conn(0x7ff28013f710 legacy=0x7ff2801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 monclient: found mon.noname-b 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800b44c0 con 0x7ff2801430c0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: _finish_auth 0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff278001a10 con 0x7ff2801430c0 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff27effd640 10 monclient: dump: 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:32.025 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff286182640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:30:32.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.022+0000 7ff286182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff278001b70 con 0x7ff2801430c0 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 1 -- 172.21.15.176:0/1804058891 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff278001f80 con 0x7ff2801430c0 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 4 set_mon_vals no callback set 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff27effd640 10 monclient: dump: 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.036 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:32.037 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:32.037 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:32.037 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:32.037 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:32.037 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 10 monclient: shutdown 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 1 -- 172.21.15.176:0/1804058891 >> v1:172.21.15.176:6790/0 conn(0x7ff2801430c0 legacy=0x7ff28014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 1 -- 172.21.15.176:0/1804058891 shutdown_connections 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.034+0000 7ff286182640 1 -- 172.21.15.176:0/1804058891 >> 172.21.15.176:0/1804058891 conn(0x7ff2800b7ea0 msgr2=0x7ff2800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.035+0000 7ff286182640 1 -- 172.21.15.176:0/1804058891 shutdown_connections 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.035+0000 7ff286182640 1 -- 172.21.15.176:0/1804058891 wait complete. 2024-03-25T18:30:32.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.035+0000 7ff286182640 10 monclient: build_initial_monmap 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.035+0000 7ff286182640 10 monclient: monmap: 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 librados: starting msgr at 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 librados: starting objecter 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 Processor -- start 2024-03-25T18:30:32.040 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 -- start start 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 librados: setting wanted keys 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 1 librados: calling monclient init 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient: init 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient(hunting): picked mon.noname-c con 0x7ff2800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient(hunting): picked mon.noname-b con 0x7ff28013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.036+0000 7ff286182640 10 monclient(hunting): picked mon.noname-a con 0x7ff2801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.037+0000 7ff286182640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2801d8340 con 0x7ff2801430c0 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.037+0000 7ff286182640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2801d7710 con 0x7ff28013f710 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.037+0000 7ff286182640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280138e90 con 0x7ff2800a98f0 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.037+0000 7ff286182640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:32.041 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.037+0000 7ff286182640 10 monclient(hunting): authenticate will time out at 8510.349888s 2024-03-25T18:30:32.042 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.040+0000 7ff277fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff28013f710 0x7ff280138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:49584/0 (socket says 172.21.15.176:49584) 2024-03-25T18:30:32.042 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.040+0000 7ff277fff640 1 -- 172.21.15.176:0/2322844352 learned_addr learned my addr 172.21.15.176:0/2322844352 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:32.042 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.040+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff28013f710 2024-03-25T18:30:32.042 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.040+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.048+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538043722 0 0) 0x7ff2801d7710 con 0x7ff28013f710 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.049+0000 7ff27cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.049+0000 7ff27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.049+0000 7ff27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.049+0000 7ff27cff9640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:30:32.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.049+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff260004eb0 con 0x7ff28013f710 2024-03-25T18:30:32.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.052+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff2801430c0 2024-03-25T18:30:32.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.052+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153239745 0 0) 0x7ff2801d8340 con 0x7ff2801430c0 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:30:32.061 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.059+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2801d7710 con 0x7ff2801430c0 2024-03-25T18:30:32.064 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917096494 0 0) 0x7ff260004eb0 con 0x7ff28013f710 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2801d8340 con 0x7ff28013f710 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff268001a30 con 0x7ff28013f710 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 10 monclient(hunting): dump: 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:32.065 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:32.066 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:32.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.062+0000 7ff27cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:32.067 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.065+0000 7ff27cff9640 10 client.?.objecter ms_handle_connect 0x7ff2800a98f0 2024-03-25T18:30:32.067 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.065+0000 7ff27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2595497584 0 0) 0x7ff2801d8340 con 0x7ff28013f710 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 >> v1:172.21.15.176:6790/0 conn(0x7ff2800a98f0 legacy=0x7ff2801d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 >> v1:172.21.15.176:6789/0 conn(0x7ff2801430c0 legacy=0x7ff280133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 1 monclient: found mon.b 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280139020 con 0x7ff28013f710 2024-03-25T18:30:32.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff27cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff286182640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff286182640 10 monclient: _renew_subs 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff286182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff280137f30 con 0x7ff28013f710 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.069+0000 7ff286182640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 10 monclient: _renew_subs 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2801352b0 con 0x7ff28013f710 2024-03-25T18:30:32.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 1 librados: init done 2024-03-25T18:30:32.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:32.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:32.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:32.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.070+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280138230 con 0x7ff28013f710 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff268001c90 con 0x7ff28013f710 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff268003a60 con 0x7ff28013f710 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27cff9640 10 monclient: dump: 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:32.357 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.354+0000 7ff27e7fc640 4 set_mon_vals no callback set 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.355+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7ff268003c40 con 0x7ff28013f710 2024-03-25T18:30:32.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.356+0000 7ff27f7fe640 10 monclient: get_auth_request con 0x7ff26006dcc0 auth_method 0 2024-03-25T18:30:32.359 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.356+0000 7ff27cff9640 10 client.4504.objecter ms_handle_connect 0x7ff26006dcc0 2024-03-25T18:30:32.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.361+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7ff268054940 con 0x7ff28013f710 2024-03-25T18:30:32.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.361+0000 7ff27cff9640 10 client.4504.objecter ms_dispatch 0x7ff280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:32.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.361+0000 7ff27cff9640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:32.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.361+0000 7ff27cff9640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:32.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.361+0000 7ff27cff9640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:30:32.366 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.364+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff268050450 con 0x7ff28013f710 2024-03-25T18:30:32.366 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.364+0000 7ff27cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:32.367 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.364+0000 7ff27cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:32.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.614+0000 7ff286182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:32.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.614+0000 7ff286182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:32.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.614+0000 7ff286182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:32.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.614+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2801d78a0 con 0x7ff28013f710 2024-03-25T18:30:32.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.638+0000 7ff27cff9640 1 -- 172.21.15.176:0/2322844352 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7ff268052380 con 0x7ff28013f710 2024-03-25T18:30:32.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.638+0000 7ff27cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:32.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.638+0000 7ff27cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:32.641 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:32.641 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:32.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 10 librados: watch_flush enter 2024-03-25T18:30:32.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 10 librados: watch_flush exit 2024-03-25T18:30:32.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:30:32.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 >> v1:172.21.15.176:6800/2598213983 conn(0x7ff26006dcc0 legacy=0x7ff260070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 10 monclient: shutdown 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 >> v1:172.21.15.186:6789/0 conn(0x7ff28013f710 legacy=0x7ff280138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 shutdown_connections 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 >> 172.21.15.176:0/2322844352 conn(0x7ff2800b7ea0 msgr2=0x7ff280074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 shutdown_connections 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 -- 172.21.15.176:0/2322844352 wait complete. 2024-03-25T18:30:32.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:32.641+0000 7ff286182640 1 librados: shutdown 2024-03-25T18:30:32.657 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:33.553 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:35.309 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.604 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:35.359 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.640 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:35.396 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.654 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:35.410 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.656 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:35.390 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.661 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:35.395 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.665 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:35.421 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.698 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:35.431 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.702 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:35.458 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:39.464 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:39.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 10 monclient: get_monmap_and_config 2024-03-25T18:30:39.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 10 monclient: build_initial_monmap 2024-03-25T18:30:39.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 10 monclient: monmap: 2024-03-25T18:30:39.581 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:39.581 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:39.580925+0000 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:39.580925+0000 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 1 Processor -- start 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 1 -- start start 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.579+0000 7fbebb6c1640 10 monclient: init 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbeb40a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:39.583 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbeb41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb4142520 con 0x7fbeb41430c0 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb40b28a0 con 0x7fbeb40a98f0 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb40b2a30 con 0x7fbeb413f710 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:39.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.580+0000 7fbebb6c1640 10 monclient(hunting): authenticate will time out at 8517.893649s 2024-03-25T18:30:39.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.586+0000 7fbeb8c5c640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fbeb413f710 0x7fbeb41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:55720/0 (socket says 172.21.15.176:55720) 2024-03-25T18:30:39.588 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.586+0000 7fbeb8c5c640 1 -- 172.21.15.176:0/3742007398 learned_addr learned my addr 172.21.15.176:0/3742007398 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:39.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772314285 0 0) 0x7fbeb40b2a30 con 0x7fbeb413f710 2024-03-25T18:30:39.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:39.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:39.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:39.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:30:39.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.596+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0003520 con 0x7fbeb413f710 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925749953 0 0) 0x7fbea0003520 con 0x7fbeb413f710 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 >> v1:172.21.15.186:6789/0 conn(0x7fbeb40a98f0 legacy=0x7fbeb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 >> v1:172.21.15.176:6789/0 conn(0x7fbeb41430c0 legacy=0x7fbeb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 monclient: found mon.noname-b 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40b44c0 con 0x7fbeb413f710 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:39.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbea4001690 con 0x7fbeb413f710 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbebb6c1640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbeabfff640 10 monclient: dump: 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:39.610 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.606+0000 7fbebb6c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:39.615 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.613+0000 7fbeabfff640 1 -- 172.21.15.176:0/3742007398 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbea40017f0 con 0x7fbeb413f710 2024-03-25T18:30:39.615 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.613+0000 7fbeabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:39.615 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.613+0000 7fbebb6c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:39.615 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.613+0000 7fbebb6c1640 4 set_mon_vals no callback set 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 10 monclient: shutdown 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 1 -- 172.21.15.176:0/3742007398 >> v1:172.21.15.176:6790/0 conn(0x7fbeb413f710 legacy=0x7fbeb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 1 -- 172.21.15.176:0/3742007398 shutdown_connections 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 1 -- 172.21.15.176:0/3742007398 >> 172.21.15.176:0/3742007398 conn(0x7fbeb40b7ea0 msgr2=0x7fbeb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 1 -- 172.21.15.176:0/3742007398 shutdown_connections 2024-03-25T18:30:39.616 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.614+0000 7fbebb6c1640 1 -- 172.21.15.176:0/3742007398 wait complete. 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 10 monclient: build_initial_monmap 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 10 monclient: monmap: 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:39.617 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 librados: starting msgr at 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 librados: starting objecter 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 Processor -- start 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 -- start start 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 librados: setting wanted keys 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 1 librados: calling monclient init 2024-03-25T18:30:39.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.615+0000 7fbebb6c1640 10 monclient: init 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbeb40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbeb413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb4138a40 con 0x7fbeb413f710 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb40ab9e0 con 0x7fbeb41430c0 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb4138d00 con 0x7fbeb40a98f0 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:39.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.616+0000 7fbebb6c1640 10 monclient(hunting): authenticate will time out at 8517.929100s 2024-03-25T18:30:39.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.620+0000 7fbeb945d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fbeb40a98f0 0x7fbeb4136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:55752/0 (socket says 172.21.15.176:55752) 2024-03-25T18:30:39.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.620+0000 7fbeb945d640 1 -- 172.21.15.176:0/1191260867 learned_addr learned my addr 172.21.15.176:0/1191260867 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:39.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.621+0000 7fbea9ffb640 10 client.?.objecter ms_handle_connect 0x7fbeb40a98f0 2024-03-25T18:30:39.623 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.621+0000 7fbea9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:39.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.630+0000 7fbea9ffb640 10 client.?.objecter ms_handle_connect 0x7fbeb413f710 2024-03-25T18:30:39.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.630+0000 7fbea9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:39.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.638+0000 7fbea9ffb640 10 client.?.objecter ms_handle_connect 0x7fbeb41430c0 2024-03-25T18:30:39.640 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.638+0000 7fbea9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:39.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528342730 0 0) 0x7fbeb40ab9e0 con 0x7fbeb41430c0 2024-03-25T18:30:39.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:39.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:39.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.643+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe98004eb0 con 0x7fbeb41430c0 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744307560 0 0) 0x7fbeb4138d00 con 0x7fbeb40a98f0 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:39.646 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:30:39.647 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.644+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb40ab9e0 con 0x7fbeb40a98f0 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438393525 0 0) 0x7fbeb4138a40 con 0x7fbeb413f710 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:30:39.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.648+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb4138d00 con 0x7fbeb413f710 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1671436655 0 0) 0x7fbeb40ab9e0 con 0x7fbeb40a98f0 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeb4138a40 con 0x7fbeb40a98f0 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbeb0001970 con 0x7fbeb40a98f0 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:39.670 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.667+0000 7fbea9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2819003146 0 0) 0x7fbe98004eb0 con 0x7fbeb41430c0 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeb40ab9e0 con 0x7fbeb41430c0 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbeac001b90 con 0x7fbeb41430c0 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.671+0000 7fbea9ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.673 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:39.674 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227248173 0 0) 0x7fbeb4138d00 con 0x7fbeb413f710 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe98004eb0 con 0x7fbeb413f710 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbea40015d0 con 0x7fbeb413f710 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.673+0000 7fbea9ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:39.675 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:39.676 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082528270 0 0) 0x7fbeb4138a40 con 0x7fbeb40a98f0 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 >> v1:172.21.15.186:6789/0 conn(0x7fbeb41430c0 legacy=0x7fbeb4138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 >> v1:172.21.15.176:6789/0 conn(0x7fbeb413f710 legacy=0x7fbeb40ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 1 monclient: found mon.c 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40abc30 con 0x7fbeb40a98f0 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:39.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbebb6c1640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbebb6c1640 10 monclient: _renew_subs 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbebb6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.685+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb4139980 con 0x7fbeb40a98f0 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.686+0000 7fbebb6c1640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.686+0000 7fbebb6c1640 10 monclient: _renew_subs 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.686+0000 7fbebb6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.686+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb41e32a0 con 0x7fbeb40a98f0 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.686+0000 7fbebb6c1640 1 librados: init done 2024-03-25T18:30:39.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.687+0000 7fbebb6c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:39.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.687+0000 7fbebb6c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:39.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.687+0000 7fbebb6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:39.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:39.687+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb40627b0 con 0x7fbeb40a98f0 2024-03-25T18:30:40.009 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbeb0003610 con 0x7fbeb40a98f0 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbeb00039a0 con 0x7fbeb40a98f0 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbea9ffb640 10 monclient: dump: 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:40.010 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.007+0000 7fbeab7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.008+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fbeb0003bc0 con 0x7fbeb40a98f0 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.009+0000 7fbeb8c5c640 10 monclient: get_auth_request con 0x7fbe9806e0f0 auth_method 0 2024-03-25T18:30:40.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.009+0000 7fbea9ffb640 10 client.4517.objecter ms_handle_connect 0x7fbe9806e0f0 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fbeb0054750 con 0x7fbeb40a98f0 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 10 client.4517.objecter ms_dispatch 0x7fbeb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbeb0050260 con 0x7fbeb40a98f0 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:40.016 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.013+0000 7fbea9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:40.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.258+0000 7fbebb6c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:40.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.258+0000 7fbebb6c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:40.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.258+0000 7fbebb6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:30:40.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.258+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbeb413df00 con 0x7fbeb40a98f0 2024-03-25T18:30:40.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.331+0000 7fbea9ffb640 1 -- 172.21.15.176:0/1191260867 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fbeb0052190 con 0x7fbeb40a98f0 2024-03-25T18:30:40.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.331+0000 7fbea9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:40.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.331+0000 7fbea9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:40.333 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:40.333 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:40.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 10 librados: watch_flush enter 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 10 librados: watch_flush exit 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 10 client.4517.objecter successfully canceled tick 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 >> v1:172.21.15.176:6800/2598213983 conn(0x7fbe9806e0f0 legacy=0x7fbe980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 10 monclient: shutdown 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 >> v1:172.21.15.176:6790/0 conn(0x7fbeb40a98f0 legacy=0x7fbeb4136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 shutdown_connections 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 >> 172.21.15.176:0/1191260867 conn(0x7fbeb40b7ea0 msgr2=0x7fbeb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 shutdown_connections 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 -- 172.21.15.176:0/1191260867 wait complete. 2024-03-25T18:30:40.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:40.333+0000 7fbebb6c1640 1 librados: shutdown 2024-03-25T18:30:40.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:47.155 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.272+0000 7f65673cf640 10 monclient: get_monmap_and_config 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.272+0000 7f65673cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.272+0000 7f65673cf640 10 monclient: monmap: 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:47.274 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:47.274307+0000 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:47.274307+0000 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.273+0000 7f65673cf640 1 Processor -- start 2024-03-25T18:30:47.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.273+0000 7f65673cf640 1 -- start start 2024-03-25T18:30:47.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.273+0000 7f65673cf640 10 monclient: init 2024-03-25T18:30:47.276 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-a con 0x7f656013fc60 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6560070570 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-c con 0x7f65600727a0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656013c9f0 con 0x7f656013fc60 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656006fac0 con 0x7f65600727a0 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600715d0 con 0x7f6560070570 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:47.277 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.274+0000 7f65673cf640 10 monclient(hunting): authenticate will time out at 8525.587296s 2024-03-25T18:30:47.290 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.289+0000 7f656516b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f656013fc60 0x7f656006f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:42294/0 (socket says 172.21.15.176:42294) 2024-03-25T18:30:47.291 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.289+0000 7f656516b640 1 -- 172.21.15.176:0/804842837 learned_addr learned my addr 172.21.15.176:0/804842837 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.312+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089807104 0 0) 0x7f656013c9f0 con 0x7f656013fc60 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.313+0000 7f6557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.313+0000 7f6557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.313+0000 7f6557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.313+0000 7f6557fff640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:30:47.315 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.313+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c003520 con 0x7f656013fc60 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715736177 0 0) 0x7f656006fac0 con 0x7f65600727a0 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:30:47.318 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.315+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656013c9f0 con 0x7f65600727a0 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061137093 0 0) 0x7f65600715d0 con 0x7f6560070570 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:30:47.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.329+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656006fac0 con 0x7f6560070570 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445081916 0 0) 0x7f656013c9f0 con 0x7f65600727a0 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 >> v1:172.21.15.176:6790/0 conn(0x7f6560070570 legacy=0x7f6560072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 >> v1:172.21.15.176:6789/0 conn(0x7f656013fc60 legacy=0x7f656006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 monclient: found mon.noname-c 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:47.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600718b0 con 0x7f65600727a0 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f65580019f0 con 0x7f65600727a0 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f65673cf640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f65673cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.332+0000 7f6557fff640 10 monclient: dump: 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:47.336 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:47.337 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:47.337 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:47.337 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:47.337 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.333+0000 7f65673cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6558001b50 con 0x7f65600727a0 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 1 -- 172.21.15.176:0/804842837 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6558001f60 con 0x7f65600727a0 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f65673cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f65673cf640 4 set_mon_vals no callback set 2024-03-25T18:30:47.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.356+0000 7f6557fff640 10 monclient: dump: 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:47.359 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.361 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 10 monclient: shutdown 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 1 -- 172.21.15.176:0/804842837 >> v1:172.21.15.186:6789/0 conn(0x7f65600727a0 legacy=0x7f656014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 1 -- 172.21.15.176:0/804842837 shutdown_connections 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 1 -- 172.21.15.176:0/804842837 >> 172.21.15.176:0/804842837 conn(0x7f65600b4a80 msgr2=0x7f65600b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 1 -- 172.21.15.176:0/804842837 shutdown_connections 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.357+0000 7f65673cf640 1 -- 172.21.15.176:0/804842837 wait complete. 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 10 monclient: monmap: 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:47.362 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 librados: starting msgr at 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 librados: starting objecter 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 Processor -- start 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 -- start start 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 librados: setting wanted keys 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 1 librados: calling monclient init 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.358+0000 7f65673cf640 10 monclient: init 2024-03-25T18:30:47.363 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6560070570 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-c con 0x7f65600727a0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient(hunting): picked mon.noname-a con 0x7f656013fc60 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600ac540 con 0x7f656013fc60 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600ac820 con 0x7f6560070570 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600ab4b0 con 0x7f65600727a0 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:47.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.359+0000 7f65673cf640 10 monclient(hunting): authenticate will time out at 8525.672400s 2024-03-25T18:30:47.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.367+0000 7f656516b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6560070570 0x7f65600ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:40958/0 (socket says 172.21.15.176:40958) 2024-03-25T18:30:47.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.367+0000 7f656516b640 1 -- 172.21.15.176:0/3937775864 learned_addr learned my addr 172.21.15.176:0/3937775864 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:47.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.370+0000 7f6555ffb640 10 client.?.objecter ms_handle_connect 0x7f6560070570 2024-03-25T18:30:47.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.370+0000 7f6555ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981567646 0 0) 0x7f65600ac820 con 0x7f6560070570 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:30:47.384 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.381+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544004eb0 con 0x7f6560070570 2024-03-25T18:30:47.386 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.384+0000 7f6555ffb640 10 client.?.objecter ms_handle_connect 0x7f656013fc60 2024-03-25T18:30:47.386 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.384+0000 7f6555ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:47.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.385+0000 7f6555ffb640 10 client.?.objecter ms_handle_connect 0x7f65600727a0 2024-03-25T18:30:47.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.385+0000 7f6555ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:47.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780056190 0 0) 0x7f6544004eb0 con 0x7f6560070570 2024-03-25T18:30:47.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65600ac820 con 0x7f6560070570 2024-03-25T18:30:47.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f655c001830 con 0x7f6560070570 2024-03-25T18:30:47.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:47.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.387+0000 7f6555ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1627235310 0 0) 0x7f65600ac820 con 0x7f6560070570 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 >> v1:172.21.15.176:6790/0 conn(0x7f65600727a0 legacy=0x7f6560141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 >> v1:172.21.15.176:6789/0 conn(0x7f656013fc60 legacy=0x7f65601dffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 1 monclient: found mon.b 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600ab1d0 con 0x7f6560070570 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f6555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 10 monclient: _renew_subs 2024-03-25T18:30:47.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65600add80 con 0x7f6560070570 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 10 monclient: _renew_subs 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65600acfb0 con 0x7f6560070570 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.398+0000 7f65673cf640 1 librados: init done 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.399+0000 7f65673cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:47.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.399+0000 7f65673cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:47.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.399+0000 7f65673cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.403 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.399+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65600ab640 con 0x7f6560070570 2024-03-25T18:30:47.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c001d20 con 0x7f6560070570 2024-03-25T18:30:47.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:47.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f655c0049c0 con 0x7f6560070570 2024-03-25T18:30:47.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:47.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f6555ffb640 10 monclient: dump: 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:47.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.617+0000 7f65577fe640 4 set_mon_vals no callback set 2024-03-25T18:30:47.673 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.671+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f655c004b20 con 0x7f6560070570 2024-03-25T18:30:47.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.672+0000 7f656496a640 10 monclient: get_auth_request con 0x7f654406dd20 auth_method 0 2024-03-25T18:30:47.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.672+0000 7f6555ffb640 10 client.4516.objecter ms_handle_connect 0x7f654406dd20 2024-03-25T18:30:47.677 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.674+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f655c054800 con 0x7f6560070570 2024-03-25T18:30:47.677 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.674+0000 7f6555ffb640 10 client.4516.objecter ms_dispatch 0x7f6560000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:47.677 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.674+0000 7f6555ffb640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:47.677 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.674+0000 7f6555ffb640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:47.677 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.675+0000 7f6555ffb640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:30:47.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.684+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f655c050140 con 0x7f6560070570 2024-03-25T18:30:47.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.684+0000 7f6555ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:47.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.684+0000 7f6555ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:47.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.906+0000 7f65673cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:47.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.906+0000 7f65673cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:47.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.906+0000 7f65673cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:47.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.906+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65600b30e0 con 0x7f6560070570 2024-03-25T18:30:47.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.949+0000 7f6555ffb640 1 -- 172.21.15.176:0/3937775864 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f655c052240 con 0x7f6560070570 2024-03-25T18:30:47.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.949+0000 7f6555ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:47.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.949+0000 7f6555ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:47.951 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:47.951 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.951+0000 7f65673cf640 10 librados: watch_flush enter 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.951+0000 7f65673cf640 10 librados: watch_flush exit 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.951+0000 7f65673cf640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.951+0000 7f65673cf640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 >> v1:172.21.15.176:6800/2598213983 conn(0x7f654406dd20 legacy=0x7f65440701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 10 monclient: shutdown 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:47.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 >> v1:172.21.15.186:6789/0 conn(0x7f6560070570 legacy=0x7f65600ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:47.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 shutdown_connections 2024-03-25T18:30:47.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 >> 172.21.15.176:0/3937775864 conn(0x7f65600b4a80 msgr2=0x7f65600afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:47.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 shutdown_connections 2024-03-25T18:30:47.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 -- 172.21.15.176:0/3937775864 wait complete. 2024-03-25T18:30:47.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:47.952+0000 7f65673cf640 1 librados: shutdown 2024-03-25T18:30:47.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:53.666 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:55.422 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.678 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:55.434 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.710 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:06:55.466 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.733 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:55.467 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.739 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:55.473 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.743 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:55.499 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.752 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:55.508 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.772 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:55.506 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.786 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:06:55.542 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.773 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.887+0000 7f2555451640 10 monclient: get_monmap_and_config 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.887+0000 7f2555451640 10 monclient: build_initial_monmap 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.887+0000 7f2555451640 10 monclient: monmap: 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:30:54.888096+0000 2024-03-25T18:30:54.888 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:30:54.888096+0000 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:30:54.889 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 1 Processor -- start 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 1 -- start start 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient: init 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient(hunting): picked mon.noname-c con 0x7f25500a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient(hunting): picked mon.noname-a con 0x7f255013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:54.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient(hunting): picked mon.noname-b con 0x7f25501430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:54.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2550142520 con 0x7f255013f710 2024-03-25T18:30:54.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500b28a0 con 0x7f25500a98f0 2024-03-25T18:30:54.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500b2a30 con 0x7f25501430c0 2024-03-25T18:30:54.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:54.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.888+0000 7f2555451640 10 monclient(hunting): authenticate will time out at 8533.200636s 2024-03-25T18:30:54.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.899+0000 7f254f7fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f25501430c0 0x7f255014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:38504/0 (socket says 172.21.15.176:38504) 2024-03-25T18:30:54.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.899+0000 7f254f7fe640 1 -- 172.21.15.176:0/2141738511 learned_addr learned my addr 172.21.15.176:0/2141738511 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498354754 0 0) 0x7f25500b2a30 con 0x7f25501430c0 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:30:54.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.919+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2538003520 con 0x7f25501430c0 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315934948 0 0) 0x7f25500b28a0 con 0x7f25500a98f0 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:30:54.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.924+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25500b2a30 con 0x7f25500a98f0 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.927+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637830162 0 0) 0x7f2538003520 con 0x7f25501430c0 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.927+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 >> v1:172.21.15.186:6789/0 conn(0x7f25500a98f0 legacy=0x7f25500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 >> v1:172.21.15.176:6789/0 conn(0x7f255013f710 legacy=0x7f25501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 1 monclient: found mon.noname-b 2024-03-25T18:30:54.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25500b44c0 con 0x7f25501430c0 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2548001a10 con 0x7f25501430c0 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f2555451640 5 monclient: authenticate success, global_id 4526 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f2555451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f254dffb640 10 monclient: dump: 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.930 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:54.931 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.928+0000 7f2555451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:54.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2548001b70 con 0x7f25501430c0 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 1 -- 172.21.15.176:0/2141738511 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2548001f80 con 0x7f25501430c0 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f2555451640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f2555451640 4 set_mon_vals no callback set 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.943+0000 7f254dffb640 10 monclient: dump: 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:54.945 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:54.946 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:54.946 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:54.946 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 10 monclient: shutdown 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 1 -- 172.21.15.176:0/2141738511 >> v1:172.21.15.176:6790/0 conn(0x7f25501430c0 legacy=0x7f255014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 1 -- 172.21.15.176:0/2141738511 shutdown_connections 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 1 -- 172.21.15.176:0/2141738511 >> 172.21.15.176:0/2141738511 conn(0x7f25500b7ea0 msgr2=0x7f25500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:54.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 1 -- 172.21.15.176:0/2141738511 shutdown_connections 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.944+0000 7f2555451640 1 -- 172.21.15.176:0/2141738511 wait complete. 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.945+0000 7f2555451640 10 monclient: build_initial_monmap 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.945+0000 7f2555451640 10 monclient: monmap: 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.945+0000 7f2555451640 1 librados: starting msgr at 2024-03-25T18:30:54.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.945+0000 7f2555451640 1 librados: starting objecter 2024-03-25T18:30:54.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.946+0000 7f2555451640 1 Processor -- start 2024-03-25T18:30:54.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.946+0000 7f2555451640 1 -- start start 2024-03-25T18:30:54.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.946+0000 7f2555451640 1 librados: setting wanted keys 2024-03-25T18:30:54.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.946+0000 7f2555451640 1 librados: calling monclient init 2024-03-25T18:30:54.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.946+0000 7f2555451640 10 monclient: init 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient(hunting): picked mon.noname-a con 0x7f25500a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient(hunting): picked mon.noname-b con 0x7f255013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient(hunting): picked mon.noname-c con 0x7f25501430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:30:54.950 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25501d8340 con 0x7f25500a98f0 2024-03-25T18:30:54.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25501d7710 con 0x7f255013f710 2024-03-25T18:30:54.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2550138e90 con 0x7f25501430c0 2024-03-25T18:30:54.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:54.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.947+0000 7f2555451640 10 monclient(hunting): authenticate will time out at 8533.259128s 2024-03-25T18:30:54.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.956+0000 7f254e7fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f255013f710 0x7f2550138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:53272/0 (socket says 172.21.15.176:53272) 2024-03-25T18:30:54.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.956+0000 7f254e7fc640 1 -- 172.21.15.176:0/847933388 learned_addr learned my addr 172.21.15.176:0/847933388 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:30:54.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.957+0000 7f25377fe640 10 client.?.objecter ms_handle_connect 0x7f255013f710 2024-03-25T18:30:54.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.957+0000 7f25377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:54.969 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.968+0000 7f25377fe640 10 client.?.objecter ms_handle_connect 0x7f25500a98f0 2024-03-25T18:30:54.969 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.968+0000 7f25377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:54.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.970+0000 7f25377fe640 10 client.?.objecter ms_handle_connect 0x7f25501430c0 2024-03-25T18:30:54.971 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.970+0000 7f25377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692112826 0 0) 0x7f25501d7710 con 0x7f255013f710 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:30:54.976 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.974+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f253c004eb0 con 0x7f255013f710 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306483750 0 0) 0x7f25501d8340 con 0x7f25500a98f0 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:30:54.983 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.982+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25501d7710 con 0x7f25500a98f0 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921365670 0 0) 0x7f2550138e90 con 0x7f25501430c0 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:30:54.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.993+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25501d8340 con 0x7f25501430c0 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 73237627 0 0) 0x7f253c004eb0 con 0x7f255013f710 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2550138e90 con 0x7f255013f710 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25440019d0 con 0x7f255013f710 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 10 monclient(hunting): dump: 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:54.998 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.996+0000 7f25377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13649929 0 0) 0x7f25501d7710 con 0x7f25500a98f0 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f253c004eb0 con 0x7f25500a98f0 2024-03-25T18:30:54.999 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f2540001930 con 0x7f25500a98f0 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:54.998+0000 7f25377fe640 10 monclient(hunting): dump: 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:55.000 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3548102434 0 0) 0x7f2550138e90 con 0x7f255013f710 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 >> v1:172.21.15.176:6790/0 conn(0x7f25501430c0 legacy=0x7f2550133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 >> v1:172.21.15.176:6789/0 conn(0x7f25500a98f0 legacy=0x7f25501d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 1 monclient: found mon.b 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:55.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550139020 con 0x7f255013f710 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f25377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f2555451640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.011+0000 7f2555451640 10 monclient: _renew_subs 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2550137f30 con 0x7f255013f710 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 10 monclient: _renew_subs 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25501352b0 con 0x7f255013f710 2024-03-25T18:30:55.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.012+0000 7f2555451640 1 librados: init done 2024-03-25T18:30:55.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.013+0000 7f2555451640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:55.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.013+0000 7f2555451640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:55.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.013+0000 7f2555451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:55.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.013+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25501d78a0 con 0x7f255013f710 2024-03-25T18:30:55.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2544003610 con 0x7f255013f710 2024-03-25T18:30:55.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:55.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f25440039a0 con 0x7f255013f710 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 10 monclient: dump: 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:30:55.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f254d7fa640 4 set_mon_vals no callback set 2024-03-25T18:30:55.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.292+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f2544003bc0 con 0x7f255013f710 2024-03-25T18:30:55.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.293+0000 7f254effd640 10 monclient: get_auth_request con 0x7f253c06df70 auth_method 0 2024-03-25T18:30:55.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.293+0000 7f25377fe640 10 client.4522.objecter ms_handle_connect 0x7f253c06df70 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.298+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f25440547e0 con 0x7f255013f710 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.298+0000 7f25377fe640 10 client.4522.objecter ms_dispatch 0x7f2550000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.298+0000 7f25377fe640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.298+0000 7f25377fe640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.298+0000 7f25377fe640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.299+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25440502f0 con 0x7f255013f710 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.299+0000 7f25377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:55.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.299+0000 7f25377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:55.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.515+0000 7f2555451640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:55.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.515+0000 7f2555451640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:55.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.515+0000 7f2555451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:30:55.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.515+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25501d7500 con 0x7f255013f710 2024-03-25T18:30:55.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.545+0000 7f25377fe640 1 -- 172.21.15.176:0/847933388 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f2544052220 con 0x7f255013f710 2024-03-25T18:30:55.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.545+0000 7f25377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:55.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.545+0000 7f25377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:55.547 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:30:55.547 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 10 librados: watch_flush enter 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 10 librados: watch_flush exit 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 >> v1:172.21.15.176:6800/2598213983 conn(0x7f253c06df70 legacy=0x7f253c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:55.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 10 monclient: shutdown 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 >> v1:172.21.15.186:6789/0 conn(0x7f255013f710 legacy=0x7f2550138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 shutdown_connections 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.547+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 >> 172.21.15.176:0/847933388 conn(0x7f25500b7ea0 msgr2=0x7f2550074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.548+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 shutdown_connections 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.548+0000 7f2555451640 1 -- 172.21.15.176:0/847933388 wait complete. 2024-03-25T18:30:55.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:30:55.548+0000 7f2555451640 1 librados: shutdown 2024-03-25T18:30:55.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:02.367 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:02.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.485+0000 7f544c590640 10 monclient: get_monmap_and_config 2024-03-25T18:31:02.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.485+0000 7f544c590640 10 monclient: build_initial_monmap 2024-03-25T18:31:02.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.485+0000 7f544c590640 10 monclient: monmap: 2024-03-25T18:31:02.486 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:02.486 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:02.486314+0000 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:02.486314+0000 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:02.487 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.486+0000 7f544c590640 1 Processor -- start 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.486+0000 7f544c590640 1 -- start start 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.486+0000 7f544c590640 10 monclient: init 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:02.488 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient(hunting): picked mon.noname-a con 0x7f544406c380 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient(hunting): picked mon.noname-c con 0x7f54440b12d0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient(hunting): picked mon.noname-b con 0x7f54440aedf0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54440b16f0 con 0x7f544406c380 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54440adfe0 con 0x7f54440b12d0 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54440add00 con 0x7f54440aedf0 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:02.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.487+0000 7f544c590640 10 monclient(hunting): authenticate will time out at 8540.799330s 2024-03-25T18:31:02.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.500+0000 7f5449b2b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f54440b12d0 0x7f54440ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:57882/0 (socket says 172.21.15.176:57882) 2024-03-25T18:31:02.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.500+0000 7f5449b2b640 1 -- 172.21.15.176:0/3632694636 learned_addr learned my addr 172.21.15.176:0/3632694636 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:02.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359303537 0 0) 0x7f54440adfe0 con 0x7f54440b12d0 2024-03-25T18:31:02.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:02.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:02.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:02.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:31:02.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.513+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5430003520 con 0x7f54440b12d0 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083445154 0 0) 0x7f54440b16f0 con 0x7f544406c380 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:31:02.524 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.523+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54440adfe0 con 0x7f544406c380 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2019044625 0 0) 0x7f5430003520 con 0x7f54440b12d0 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 >> v1:172.21.15.176:6790/0 conn(0x7f54440aedf0 legacy=0x7f544414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 >> v1:172.21.15.176:6789/0 conn(0x7f544406c380 legacy=0x7f54440b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: _finish_hunting 0 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 monclient: found mon.noname-c 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440ae2c0 con 0x7f54440b12d0 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: _finish_auth 0 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: _check_auth_tickets 2024-03-25T18:31:02.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5434001690 con 0x7f54440b12d0 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544932a640 10 monclient: dump: 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544c590640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:31:02.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.524+0000 7f544c590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:02.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.529+0000 7f544932a640 1 -- 172.21.15.176:0/3632694636 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54340017f0 con 0x7f54440b12d0 2024-03-25T18:31:02.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.529+0000 7f544932a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:02.530 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.529+0000 7f544c590640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:02.531 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.529+0000 7f544c590640 4 set_mon_vals no callback set 2024-03-25T18:31:02.531 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 10 monclient: shutdown 2024-03-25T18:31:02.531 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:02.531 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 1 -- 172.21.15.176:0/3632694636 >> v1:172.21.15.186:6789/0 conn(0x7f54440b12d0 legacy=0x7f54440ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 1 -- 172.21.15.176:0/3632694636 shutdown_connections 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 1 -- 172.21.15.176:0/3632694636 >> 172.21.15.176:0/3632694636 conn(0x7f54440b5d10 msgr2=0x7f54440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 1 -- 172.21.15.176:0/3632694636 shutdown_connections 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.530+0000 7f544c590640 1 -- 172.21.15.176:0/3632694636 wait complete. 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.531+0000 7f544c590640 10 monclient: build_initial_monmap 2024-03-25T18:31:02.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.531+0000 7f544c590640 10 monclient: monmap: 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.531+0000 7f544c590640 1 librados: starting msgr at 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.531+0000 7f544c590640 1 librados: starting objecter 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.531+0000 7f544c590640 1 Processor -- start 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 -- start start 2024-03-25T18:31:02.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 librados: setting wanted keys 2024-03-25T18:31:02.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 librados: calling monclient init 2024-03-25T18:31:02.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient: init 2024-03-25T18:31:02.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient(hunting): picked mon.noname-a con 0x7f544406c380 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient(hunting): picked mon.noname-c con 0x7f54440aedf0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient(hunting): picked mon.noname-b con 0x7f54440b12d0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406c8e0 con 0x7f544406c380 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54441a5570 con 0x7f54440b12d0 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54441a3be0 con 0x7f54440aedf0 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:02.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.532+0000 7f544c590640 10 monclient(hunting): authenticate will time out at 8540.844607s 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.542+0000 7f544a32c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f544406c380 0x7f54441d7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:39850/0 (socket says 172.21.15.176:39850) 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.542+0000 7f544a32c640 1 -- 172.21.15.176:0/2871866436 learned_addr learned my addr 172.21.15.176:0/2871866436 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.542+0000 7f543affd640 10 client.?.objecter ms_handle_connect 0x7f544406c380 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.542+0000 7f543affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.545+0000 7f543affd640 10 client.?.objecter ms_handle_connect 0x7f54440aedf0 2024-03-25T18:31:02.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.545+0000 7f543affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:02.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.546+0000 7f543affd640 10 client.?.objecter ms_handle_connect 0x7f54440b12d0 2024-03-25T18:31:02.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.546+0000 7f543affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779800708 0 0) 0x7f54441a3be0 con 0x7f54440aedf0 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:31:02.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.555+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428004eb0 con 0x7f54440aedf0 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117934249 0 0) 0x7f544406c8e0 con 0x7f544406c380 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:31:02.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.559+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54441a3be0 con 0x7f544406c380 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048743394 0 0) 0x7f54441a5570 con 0x7f54440b12d0 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:31:02.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.560+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544406c8e0 con 0x7f54440b12d0 2024-03-25T18:31:02.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938758032 0 0) 0x7f54441a3be0 con 0x7f544406c380 2024-03-25T18:31:02.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54441a5570 con 0x7f544406c380 2024-03-25T18:31:02.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5440001a70 con 0x7f544406c380 2024-03-25T18:31:02.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 10 monclient(hunting): dump: 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.576+0000 7f543affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373022468 0 0) 0x7f5428004eb0 con 0x7f54440aedf0 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54441a3be0 con 0x7f54440aedf0 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5434001b00 con 0x7f54440aedf0 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:02.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.579+0000 7f543affd640 10 monclient(hunting): dump: 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:02.581 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418282256 0 0) 0x7f544406c8e0 con 0x7f54440b12d0 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5428004eb0 con 0x7f54440b12d0 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f543c001b00 con 0x7f54440b12d0 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:02.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.591+0000 7f543affd640 10 monclient(hunting): dump: 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:02.593 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1631817360 0 0) 0x7f54441a5570 con 0x7f544406c380 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 >> v1:172.21.15.176:6790/0 conn(0x7f54440aedf0 legacy=0x7f54441d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 >> v1:172.21.15.186:6789/0 conn(0x7f54440b12d0 legacy=0x7f54441a34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 1 monclient: found mon.a 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54441a3d70 con 0x7f544406c380 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 10 monclient: _finish_auth 0 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f543affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 5 monclient: authenticate success, global_id 4515 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 10 monclient: _renew_subs 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:02.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54441a3fe0 con 0x7f544406c380 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 10 monclient: _renew_subs 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54441e33b0 con 0x7f544406c380 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.602+0000 7f544c590640 1 librados: init done 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.603+0000 7f544c590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:02.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.603+0000 7f544c590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:02.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.603+0000 7f544c590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:02.606 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.603+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54440627b0 con 0x7f544406c380 2024-03-25T18:31:02.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5440001c50 con 0x7f544406c380 2024-03-25T18:31:02.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:02.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5440003a60 con 0x7f544406c380 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f543affd640 10 monclient: dump: 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:02.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.901+0000 7f5448b29640 4 set_mon_vals no callback set 2024-03-25T18:31:02.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.902+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f5440003c80 con 0x7f544406c380 2024-03-25T18:31:02.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.902+0000 7f5449b2b640 10 monclient: get_auth_request con 0x7f542806e0f0 auth_method 0 2024-03-25T18:31:02.904 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.903+0000 7f543affd640 10 client.4515.objecter ms_handle_connect 0x7f542806e0f0 2024-03-25T18:31:02.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f5440054870 con 0x7f544406c380 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 10 client.4515.objecter ms_dispatch 0x7f5444000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5440050380 con 0x7f544406c380 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:02.909 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:02.907+0000 7f543affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:03.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.143+0000 7f544c590640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:03.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.143+0000 7f544c590640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:03.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.143+0000 7f544c590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:03.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.143+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54440ae810 con 0x7f544406c380 2024-03-25T18:31:03.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.174+0000 7f543affd640 1 -- 172.21.15.176:0/2871866436 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f54400522b0 con 0x7f544406c380 2024-03-25T18:31:03.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.174+0000 7f543affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:03.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.174+0000 7f543affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:03.176 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:03.176 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:03.178 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 10 librados: watch_flush enter 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 10 librados: watch_flush exit 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 10 client.4515.objecter successfully canceled tick 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 >> v1:172.21.15.176:6800/2598213983 conn(0x7f542806e0f0 legacy=0x7f54280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 10 monclient: shutdown 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 >> v1:172.21.15.176:6789/0 conn(0x7f544406c380 legacy=0x7f54441d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 shutdown_connections 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 >> 172.21.15.176:0/2871866436 conn(0x7f54440b5d10 msgr2=0x7f54441c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 shutdown_connections 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 -- 172.21.15.176:0/2871866436 wait complete. 2024-03-25T18:31:03.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:03.177+0000 7f544c590640 1 librados: shutdown 2024-03-25T18:31:03.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:09.999 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:10.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.110+0000 7f6017833640 10 monclient: get_monmap_and_config 2024-03-25T18:31:10.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.110+0000 7f6017833640 10 monclient: build_initial_monmap 2024-03-25T18:31:10.111 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.110+0000 7f6017833640 10 monclient: monmap: 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:10.111181+0000 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:10.111181+0000 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:10.112 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.111+0000 7f6017833640 1 Processor -- start 2024-03-25T18:31:10.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.111+0000 7f6017833640 1 -- start start 2024-03-25T18:31:10.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.111+0000 7f6017833640 10 monclient: init 2024-03-25T18:31:10.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.111+0000 7f6017833640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:10.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.111+0000 7f6017833640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 10 monclient(hunting): picked mon.noname-c con 0x7f60100a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 10 monclient(hunting): picked mon.noname-b con 0x7f601013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 10 monclient(hunting): picked mon.noname-a con 0x7f60101430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6010142520 con 0x7f60101430c0 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100b28a0 con 0x7f60100a98f0 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100b2a30 con 0x7f601013f710 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:10.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.112+0000 7f6017833640 10 monclient(hunting): authenticate will time out at 8548.423917s 2024-03-25T18:31:10.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.125+0000 7f60155cf640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f60100a98f0 0x7f60100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:57796/0 (socket says 172.21.15.176:57796) 2024-03-25T18:31:10.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.125+0000 7f60155cf640 1 -- 172.21.15.176:0/2463452375 learned_addr learned my addr 172.21.15.176:0/2463452375 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622477250 0 0) 0x7f6010142520 con 0x7f60101430c0 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:31:10.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.141+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc003520 con 0x7f60101430c0 2024-03-25T18:31:10.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288146093 0 0) 0x7f60100b28a0 con 0x7f60100a98f0 2024-03-25T18:31:10.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:10.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:10.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:10.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:31:10.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.146+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6010142520 con 0x7f60100a98f0 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267497467 0 0) 0x7f5ffc003520 con 0x7f60101430c0 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 >> v1:172.21.15.176:6790/0 conn(0x7f601013f710 legacy=0x7f60101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 >> v1:172.21.15.186:6789/0 conn(0x7f60100a98f0 legacy=0x7f60100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 monclient: found mon.noname-a 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100b44c0 con 0x7f60101430c0 2024-03-25T18:31:10.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6008001a10 con 0x7f60101430c0 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6007fff640 10 monclient: dump: 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:10.154 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:10.155 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:10.155 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6017833640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:31:10.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.151+0000 7f6017833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:10.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6008001b70 con 0x7f60101430c0 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 1 -- 172.21.15.176:0/2463452375 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6008001f80 con 0x7f60101430c0 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6017833640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6017833640 4 set_mon_vals no callback set 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.162+0000 7f6007fff640 10 monclient: dump: 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:10.164 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.165 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:10.165 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:10.165 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:10.165 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 10 monclient: shutdown 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 1 -- 172.21.15.176:0/2463452375 >> v1:172.21.15.176:6789/0 conn(0x7f60101430c0 legacy=0x7f601014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 1 -- 172.21.15.176:0/2463452375 shutdown_connections 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 1 -- 172.21.15.176:0/2463452375 >> 172.21.15.176:0/2463452375 conn(0x7f60100b7ea0 msgr2=0x7f60100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 1 -- 172.21.15.176:0/2463452375 shutdown_connections 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.163+0000 7f6017833640 1 -- 172.21.15.176:0/2463452375 wait complete. 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.164+0000 7f6017833640 10 monclient: build_initial_monmap 2024-03-25T18:31:10.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.164+0000 7f6017833640 10 monclient: monmap: 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 librados: starting msgr at 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 librados: starting objecter 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 Processor -- start 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 -- start start 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 librados: setting wanted keys 2024-03-25T18:31:10.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 1 librados: calling monclient init 2024-03-25T18:31:10.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.165+0000 7f6017833640 10 monclient: init 2024-03-25T18:31:10.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient(hunting): picked mon.noname-a con 0x7f60100a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient(hunting): picked mon.noname-b con 0x7f601013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient(hunting): picked mon.noname-c con 0x7f60101430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60101d7df0 con 0x7f60100a98f0 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60101d7f80 con 0x7f601013f710 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601013b5c0 con 0x7f60101430c0 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:10.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.166+0000 7f6017833640 10 monclient(hunting): authenticate will time out at 8548.478047s 2024-03-25T18:31:10.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.173+0000 7f6015dd0640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f60101430c0 0x7f6010135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:45796/0 (socket says 172.21.15.176:45796) 2024-03-25T18:31:10.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.173+0000 7f6015dd0640 1 -- 172.21.15.176:0/552287377 learned_addr learned my addr 172.21.15.176:0/552287377 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:10.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.175+0000 7f6005ffb640 10 client.?.objecter ms_handle_connect 0x7f60101430c0 2024-03-25T18:31:10.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.175+0000 7f6005ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:10.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.179+0000 7f6005ffb640 10 client.?.objecter ms_handle_connect 0x7f601013f710 2024-03-25T18:31:10.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.179+0000 7f6005ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:10.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531701831 0 0) 0x7f601013b5c0 con 0x7f60101430c0 2024-03-25T18:31:10.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:10.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:10.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:10.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:31:10.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.184+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff4004eb0 con 0x7f60101430c0 2024-03-25T18:31:10.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.187+0000 7f6005ffb640 10 client.?.objecter ms_handle_connect 0x7f60100a98f0 2024-03-25T18:31:10.188 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.187+0000 7f6005ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764670742 0 0) 0x7f60101d7df0 con 0x7f60100a98f0 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:31:10.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.192+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f601013b5c0 con 0x7f60100a98f0 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534362874 0 0) 0x7f5ff4004eb0 con 0x7f60101430c0 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60101d7df0 con 0x7f60101430c0 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6008001f80 con 0x7f60101430c0 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.199 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:10.200 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:10.200 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:10.200 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.200 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.197+0000 7f6005ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888179672 0 0) 0x7f601013b5c0 con 0x7f60100a98f0 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ff4004eb0 con 0x7f60100a98f0 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f600c001930 con 0x7f60100a98f0 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.201+0000 7f6005ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:10.202 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:10.203 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:10.203 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.205+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637300596 0 0) 0x7f60101d7f80 con 0x7f601013f710 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.205+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.205+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.205+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.205+0000 7f6005ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:31:10.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.206+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f601013b5c0 con 0x7f601013f710 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 117402530 0 0) 0x7f5ff4004eb0 con 0x7f60100a98f0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 >> v1:172.21.15.176:6790/0 conn(0x7f60101430c0 legacy=0x7f6010135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 >> v1:172.21.15.186:6789/0 conn(0x7f601013f710 legacy=0x7f6010133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 1 monclient: found mon.a 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601013b750 con 0x7f60100a98f0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 10 monclient: _renew_subs 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6010133710 con 0x7f60100a98f0 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 10 monclient: _renew_subs 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60101357a0 con 0x7f60100a98f0 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.211+0000 7f6017833640 1 librados: init done 2024-03-25T18:31:10.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.212+0000 7f6017833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:10.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.212+0000 7f6017833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:10.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.212+0000 7f6017833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.212+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010138fd0 con 0x7f60100a98f0 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.516+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f600c001d40 con 0x7f60100a98f0 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f600c0047e0 con 0x7f60100a98f0 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 10 monclient: dump: 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:10.518 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:10.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f60077fe640 4 set_mon_vals no callback set 2024-03-25T18:31:10.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.517+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f600c004a00 con 0x7f60100a98f0 2024-03-25T18:31:10.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.518+0000 7f6014dce640 10 monclient: get_auth_request con 0x7f5ff406df30 auth_method 0 2024-03-25T18:31:10.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.518+0000 7f6005ffb640 10 client.4521.objecter ms_handle_connect 0x7f5ff406df30 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f600c0545e0 con 0x7f60100a98f0 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 10 client.4521.objecter ms_dispatch 0x7f6010000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f600c04ff20 con 0x7f60100a98f0 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:10.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.526+0000 7f6005ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:10.738 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.737+0000 7f6017833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:10.738 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.737+0000 7f6017833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:10.739 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.737+0000 7f6017833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:10.739 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.737+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6010074cc0 con 0x7f60100a98f0 2024-03-25T18:31:10.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.794+0000 7f6005ffb640 1 -- 172.21.15.176:0/552287377 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f600c052020 con 0x7f60100a98f0 2024-03-25T18:31:10.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.794+0000 7f6005ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:10.795 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.794+0000 7f6005ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:10.795 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:10.795 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 10 librados: watch_flush enter 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 10 librados: watch_flush exit 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 >> v1:172.21.15.176:6800/2598213983 conn(0x7f5ff406df30 legacy=0x7f5ff40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 10 monclient: shutdown 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 >> v1:172.21.15.176:6789/0 conn(0x7f60100a98f0 legacy=0x7f60101d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:10.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 shutdown_connections 2024-03-25T18:31:10.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 >> 172.21.15.176:0/552287377 conn(0x7f60100b7ea0 msgr2=0x7f6010074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:10.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 shutdown_connections 2024-03-25T18:31:10.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 -- 172.21.15.176:0/552287377 wait complete. 2024-03-25T18:31:10.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:10.796+0000 7f6017833640 1 librados: shutdown 2024-03-25T18:31:10.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:13.719 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:15.474 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.725 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:15.481 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.760 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:15.516 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.775 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:15.509 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.782 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:15.538 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.783 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:15.517 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.789 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:15.545 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.814 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:15.548 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.826 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:15.582 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:17.618 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.731+0000 7f416767b640 10 monclient: get_monmap_and_config 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.731+0000 7f416767b640 10 monclient: build_initial_monmap 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.731+0000 7f416767b640 10 monclient: monmap: 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:17.732629+0000 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:17.732629+0000 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:17.733 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.734 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.732+0000 7f416767b640 1 Processor -- start 2024-03-25T18:31:17.734 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.733+0000 7f416767b640 1 -- start start 2024-03-25T18:31:17.734 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.733+0000 7f416767b640 10 monclient: init 2024-03-25T18:31:17.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:17.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:17.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient(hunting): picked mon.noname-c con 0x7f41600705b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:17.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient(hunting): picked mon.noname-a con 0x7f41601402d0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient(hunting): picked mon.noname-b con 0x7f4160149ae0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41600718c0 con 0x7f41601402d0 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41600714c0 con 0x7f41600705b0 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4160072b50 con 0x7f4160149ae0 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:17.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.734+0000 7f416767b640 10 monclient(hunting): authenticate will time out at 8556.046080s 2024-03-25T18:31:17.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.741+0000 7f4164c16640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f41601402d0 0x7f4160140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:33442/0 (socket says 172.21.15.176:33442) 2024-03-25T18:31:17.742 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.741+0000 7f4164c16640 1 -- 172.21.15.176:0/2910152041 learned_addr learned my addr 172.21.15.176:0/2910152041 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:17.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309881166 0 0) 0x7f41600718c0 con 0x7f41601402d0 2024-03-25T18:31:17.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.755 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.755 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.755 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:31:17.755 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.753+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f414c003520 con 0x7f41601402d0 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144787157 0 0) 0x7f41600714c0 con 0x7f41600705b0 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:31:17.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.756+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41600718c0 con 0x7f41600705b0 2024-03-25T18:31:17.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155423994 0 0) 0x7f4160072b50 con 0x7f4160149ae0 2024-03-25T18:31:17.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:31:17.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.770+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41600714c0 con 0x7f4160149ae0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031329175 0 0) 0x7f41600718c0 con 0x7f41600705b0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 >> v1:172.21.15.176:6790/0 conn(0x7f4160149ae0 legacy=0x7f416014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 >> v1:172.21.15.176:6789/0 conn(0x7f41601402d0 legacy=0x7f4160140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 monclient: found mon.noname-c 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416013dd10 con 0x7f41600705b0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:17.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f415c001670 con 0x7f41600705b0 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f416767b640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f416767b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f4157fff640 10 monclient: dump: 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:17.780 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:17.781 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:17.781 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:17.781 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.777+0000 7f416767b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:17.807 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.805+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f415c0017d0 con 0x7f41600705b0 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f4157fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f4157fff640 1 -- 172.21.15.176:0/2910152041 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f415c001be0 con 0x7f41600705b0 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f4157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f4157fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f4157fff640 10 monclient: dump: 2024-03-25T18:31:17.812 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f416767b640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:17.813 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f416767b640 4 set_mon_vals no callback set 2024-03-25T18:31:17.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f416767b640 10 monclient: shutdown 2024-03-25T18:31:17.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f416767b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:17.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.806+0000 7f416767b640 1 -- 172.21.15.176:0/2910152041 >> v1:172.21.15.186:6789/0 conn(0x7f41600705b0 legacy=0x7f416013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:17.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 1 -- 172.21.15.176:0/2910152041 shutdown_connections 2024-03-25T18:31:17.815 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 1 -- 172.21.15.176:0/2910152041 >> 172.21.15.176:0/2910152041 conn(0x7f41600b5d10 msgr2=0x7f41600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 1 -- 172.21.15.176:0/2910152041 shutdown_connections 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 1 -- 172.21.15.176:0/2910152041 wait complete. 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 10 monclient: build_initial_monmap 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.807+0000 7f416767b640 10 monclient: monmap: 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 librados: starting msgr at 2024-03-25T18:31:17.816 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 librados: starting objecter 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 Processor -- start 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 -- start start 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 librados: setting wanted keys 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 1 librados: calling monclient init 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 10 monclient: init 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.808+0000 7f416767b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient(hunting): picked mon.noname-c con 0x7f41600705b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient(hunting): picked mon.noname-a con 0x7f41601402d0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient(hunting): picked mon.noname-b con 0x7f4160149ae0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41601421c0 con 0x7f41601402d0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4160136250 con 0x7f4160149ae0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4160137990 con 0x7f41600705b0 2024-03-25T18:31:17.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:17.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.809+0000 7f416767b640 10 monclient(hunting): authenticate will time out at 8556.120993s 2024-03-25T18:31:17.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.814+0000 7f4165c18640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4160149ae0 0x7f41601d5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:55780/0 (socket says 172.21.15.176:55780) 2024-03-25T18:31:17.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.814+0000 7f4165c18640 1 -- 172.21.15.176:0/4290380357 learned_addr learned my addr 172.21.15.176:0/4290380357 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:17.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.815+0000 7f4155ffb640 10 client.?.objecter ms_handle_connect 0x7f4160149ae0 2024-03-25T18:31:17.818 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.815+0000 7f4155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:17.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.823+0000 7f4155ffb640 10 client.?.objecter ms_handle_connect 0x7f41601402d0 2024-03-25T18:31:17.824 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.823+0000 7f4155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:17.825 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.824+0000 7f4155ffb640 10 client.?.objecter ms_handle_connect 0x7f41600705b0 2024-03-25T18:31:17.825 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.824+0000 7f4155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821013033 0 0) 0x7f4160136250 con 0x7f4160149ae0 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:31:17.827 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.826+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144004eb0 con 0x7f4160149ae0 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338633066 0 0) 0x7f41601421c0 con 0x7f41601402d0 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:31:17.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.828+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4160136250 con 0x7f41601402d0 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040798721 0 0) 0x7f4160137990 con 0x7f41600705b0 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:31:17.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.832+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41601421c0 con 0x7f41600705b0 2024-03-25T18:31:17.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248551352 0 0) 0x7f4160136250 con 0x7f41601402d0 2024-03-25T18:31:17.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4160137990 con 0x7f41601402d0 2024-03-25T18:31:17.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4150001950 con 0x7f41601402d0 2024-03-25T18:31:17.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.839+0000 7f4155ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212267106 0 0) 0x7f4144004eb0 con 0x7f4160149ae0 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4160136250 con 0x7f4160149ae0 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4158001c30 con 0x7f4160149ae0 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.846+0000 7f4155ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:17.847 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:17.848 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:17.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1328795583 0 0) 0x7f4160137990 con 0x7f41601402d0 2024-03-25T18:31:17.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 >> v1:172.21.15.176:6790/0 conn(0x7f41600705b0 legacy=0x7f4160135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 >> v1:172.21.15.186:6789/0 conn(0x7f4160149ae0 legacy=0x7f41601d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 1 monclient: found mon.a 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4160137b20 con 0x7f41601402d0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f4155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 10 monclient: _renew_subs 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4160134ea0 con 0x7f41601402d0 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 10 monclient: _renew_subs 2024-03-25T18:31:17.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4160134690 con 0x7f41601402d0 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.855+0000 7f416767b640 1 librados: init done 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.856+0000 7f416767b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.856+0000 7f416767b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.856+0000 7f416767b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:17.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:17.856+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41600627b0 con 0x7f41601402d0 2024-03-25T18:31:18.167 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4150001d60 con 0x7f41601402d0 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4150004980 con 0x7f41601402d0 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f4155ffb640 10 monclient: dump: 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:18.168 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.166+0000 7f41577fe640 4 set_mon_vals no callback set 2024-03-25T18:31:18.169 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.167+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f4150004ba0 con 0x7f41601402d0 2024-03-25T18:31:18.170 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.167+0000 7f4165417640 10 monclient: get_auth_request con 0x7f414406df20 auth_method 0 2024-03-25T18:31:18.170 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.168+0000 7f4155ffb640 10 client.4527.objecter ms_handle_connect 0x7f414406df20 2024-03-25T18:31:18.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.173+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f4150054740 con 0x7f41601402d0 2024-03-25T18:31:18.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.173+0000 7f4155ffb640 10 client.4527.objecter ms_dispatch 0x7f4160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:18.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.173+0000 7f4155ffb640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:18.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.173+0000 7f4155ffb640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:18.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.174+0000 7f4155ffb640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:31:18.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.174+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4150050080 con 0x7f41601402d0 2024-03-25T18:31:18.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.174+0000 7f4155ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:18.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.174+0000 7f4155ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:18.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.434+0000 7f416767b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:18.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.434+0000 7f416767b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:18.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.434+0000 7f416767b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:18.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.434+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41601d5380 con 0x7f41601402d0 2024-03-25T18:31:18.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.474+0000 7f4155ffb640 1 -- 172.21.15.176:0/4290380357 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f4150052180 con 0x7f41601402d0 2024-03-25T18:31:18.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.474+0000 7f4155ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:18.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.474+0000 7f4155ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:18.476 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:18.476 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:18.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 10 librados: watch_flush enter 2024-03-25T18:31:18.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 10 librados: watch_flush exit 2024-03-25T18:31:18.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:31:18.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:31:18.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 >> v1:172.21.15.176:6800/2598213983 conn(0x7f414406df20 legacy=0x7f41440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 10 monclient: shutdown 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 >> v1:172.21.15.176:6789/0 conn(0x7f41601402d0 legacy=0x7f4160133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 shutdown_connections 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 >> 172.21.15.176:0/4290380357 conn(0x7f41600b5d10 msgr2=0x7f416013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 shutdown_connections 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 -- 172.21.15.176:0/4290380357 wait complete. 2024-03-25T18:31:18.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:18.477+0000 7f416767b640 1 librados: shutdown 2024-03-25T18:31:18.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:25.297 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.413+0000 7fbe7231c640 10 monclient: get_monmap_and_config 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.413+0000 7fbe7231c640 10 monclient: build_initial_monmap 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.413+0000 7fbe7231c640 10 monclient: monmap: 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:25.414726+0000 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:25.414726+0000 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:25.415 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:25.416 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:25.416 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:25.416 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:25.416 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.414+0000 7fbe7231c640 1 Processor -- start 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.414+0000 7fbe7231c640 1 -- start start 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.414+0000 7fbe7231c640 10 monclient: init 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6c0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-c con 0x7fbe6c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6c1430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:25.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.415+0000 7fbe7231c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c142520 con 0x7fbe6c0a98f0 2024-03-25T18:31:25.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.416+0000 7fbe7231c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0b28a0 con 0x7fbe6c13f710 2024-03-25T18:31:25.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.416+0000 7fbe7231c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0b2a30 con 0x7fbe6c1430c0 2024-03-25T18:31:25.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.416+0000 7fbe7231c640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:25.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.416+0000 7fbe7231c640 10 monclient(hunting): authenticate will time out at 8563.727894s 2024-03-25T18:31:25.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.425+0000 7fbe708b9640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fbe6c1430c0 0x7fbe6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:48340/0 (socket says 172.21.15.176:48340) 2024-03-25T18:31:25.426 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.425+0000 7fbe708b9640 1 -- 172.21.15.176:0/3592085131 learned_addr learned my addr 172.21.15.176:0/3592085131 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887509883 0 0) 0x7fbe6c0b28a0 con 0x7fbe6c13f710 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:31:25.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.438+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54003520 con 0x7fbe6c13f710 2024-03-25T18:31:25.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699384365 0 0) 0x7fbe6c0b2a30 con 0x7fbe6c1430c0 2024-03-25T18:31:25.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:25.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:25.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:25.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:31:25.441 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.439+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe6c0b28a0 con 0x7fbe6c1430c0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949457590 0 0) 0x7fbe6c0b28a0 con 0x7fbe6c1430c0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 >> v1:172.21.15.186:6789/0 conn(0x7fbe6c13f710 legacy=0x7fbe6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 >> v1:172.21.15.176:6789/0 conn(0x7fbe6c0a98f0 legacy=0x7fbe6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 monclient: found mon.noname-b 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c0b44c0 con 0x7fbe6c1430c0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: _finish_auth 0 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:25.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbe640019f0 con 0x7fbe6c1430c0 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe6affd640 10 monclient: dump: 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe7231c640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:31:25.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.460+0000 7fbe7231c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:25.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe64001b50 con 0x7fbe6c1430c0 2024-03-25T18:31:25.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 1 -- 172.21.15.176:0/3592085131 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbe64001fa0 con 0x7fbe6c1430c0 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe6affd640 10 monclient: dump: 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe7231c640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:25.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.472+0000 7fbe7231c640 4 set_mon_vals no callback set 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 10 monclient: shutdown 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 1 -- 172.21.15.176:0/3592085131 >> v1:172.21.15.176:6790/0 conn(0x7fbe6c1430c0 legacy=0x7fbe6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 1 -- 172.21.15.176:0/3592085131 shutdown_connections 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 1 -- 172.21.15.176:0/3592085131 >> 172.21.15.176:0/3592085131 conn(0x7fbe6c0b7ea0 msgr2=0x7fbe6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 1 -- 172.21.15.176:0/3592085131 shutdown_connections 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.473+0000 7fbe7231c640 1 -- 172.21.15.176:0/3592085131 wait complete. 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.474+0000 7fbe7231c640 10 monclient: build_initial_monmap 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.474+0000 7fbe7231c640 10 monclient: monmap: 2024-03-25T18:31:25.476 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.474+0000 7fbe7231c640 1 librados: starting msgr at 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.474+0000 7fbe7231c640 1 librados: starting objecter 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.474+0000 7fbe7231c640 1 Processor -- start 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 -- start start 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 librados: setting wanted keys 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 librados: calling monclient init 2024-03-25T18:31:25.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient: init 2024-03-25T18:31:25.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-c con 0x7fbe6c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6c13f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6c1430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c137e20 con 0x7fbe6c13f710 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0b1720 con 0x7fbe6c1430c0 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c138710 con 0x7fbe6c0a98f0 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:25.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.475+0000 7fbe7231c640 10 monclient(hunting): authenticate will time out at 8563.787664s 2024-03-25T18:31:25.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.490+0000 7fbe6b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fbe6c13f710 0x7fbe6c1da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:53612/0 (socket says 172.21.15.176:53612) 2024-03-25T18:31:25.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.490+0000 7fbe6b7fe640 1 -- 172.21.15.176:0/536940700 learned_addr learned my addr 172.21.15.176:0/536940700 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:25.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.491+0000 7fbe68ff9640 10 client.?.objecter ms_handle_connect 0x7fbe6c13f710 2024-03-25T18:31:25.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.491+0000 7fbe68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:25.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.495+0000 7fbe68ff9640 10 client.?.objecter ms_handle_connect 0x7fbe6c0a98f0 2024-03-25T18:31:25.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.495+0000 7fbe68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:25.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.500+0000 7fbe68ff9640 10 client.?.objecter ms_handle_connect 0x7fbe6c1430c0 2024-03-25T18:31:25.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.500+0000 7fbe68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673071231 0 0) 0x7fbe6c137e20 con 0x7fbe6c13f710 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:31:25.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.501+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe58004eb0 con 0x7fbe6c13f710 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057764556 0 0) 0x7fbe58004eb0 con 0x7fbe6c13f710 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe6c137e20 con 0x7fbe6c13f710 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbe60001990 con 0x7fbe6c13f710 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:25.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.519+0000 7fbe68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:25.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482775166 0 0) 0x7fbe6c137e20 con 0x7fbe6c13f710 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 >> v1:172.21.15.176:6790/0 conn(0x7fbe6c0a98f0 legacy=0x7fbe6c137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 >> v1:172.21.15.186:6789/0 conn(0x7fbe6c1430c0 legacy=0x7fbe6c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 1 monclient: found mon.a 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c138040 con 0x7fbe6c13f710 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 5 monclient: authenticate success, global_id 4533 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 10 monclient: _renew_subs 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:25.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6c1382f0 con 0x7fbe6c13f710 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 10 monclient: _renew_subs 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6c1e2fb0 con 0x7fbe6c13f710 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.584+0000 7fbe7231c640 1 librados: init done 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.585+0000 7fbe7231c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.585+0000 7fbe7231c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.585+0000 7fbe7231c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:25.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.585+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6c0b3ef0 con 0x7fbe6c13f710 2024-03-25T18:31:25.935 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe60001c50 con 0x7fbe6c13f710 2024-03-25T18:31:25.935 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:25.935 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fbe60003a60 con 0x7fbe6c13f710 2024-03-25T18:31:25.935 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 10 monclient: dump: 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:25.936 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe6a7fc640 4 set_mon_vals no callback set 2024-03-25T18:31:25.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.934+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fbe60003c80 con 0x7fbe6c13f710 2024-03-25T18:31:25.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.935+0000 7fbe6bfff640 10 monclient: get_auth_request con 0x7fbe5806dce0 auth_method 0 2024-03-25T18:31:25.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.935+0000 7fbe68ff9640 10 client.4533.objecter ms_handle_connect 0x7fbe5806dce0 2024-03-25T18:31:25.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fbe60054800 con 0x7fbe6c13f710 2024-03-25T18:31:25.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 10 client.4533.objecter ms_dispatch 0x7fbe6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe60050310 con 0x7fbe6c13f710 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:25.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:25.941+0000 7fbe68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:26.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.189+0000 7fbe7231c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:26.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.189+0000 7fbe7231c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:26.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.189+0000 7fbe7231c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:26.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.189+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe6c0627b0 con 0x7fbe6c13f710 2024-03-25T18:31:26.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.223+0000 7fbe68ff9640 1 -- 172.21.15.176:0/536940700 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fbe60052240 con 0x7fbe6c13f710 2024-03-25T18:31:26.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.223+0000 7fbe68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:26.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.223+0000 7fbe68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:26.225 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:26.225 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 10 librados: watch_flush enter 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 10 librados: watch_flush exit 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 10 client.4533.objecter successfully canceled tick 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 >> v1:172.21.15.176:6800/2598213983 conn(0x7fbe5806dce0 legacy=0x7fbe580701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 10 monclient: shutdown 2024-03-25T18:31:26.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.225+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 >> v1:172.21.15.176:6789/0 conn(0x7fbe6c13f710 legacy=0x7fbe6c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.226+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 shutdown_connections 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.226+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 >> 172.21.15.176:0/536940700 conn(0x7fbe6c0b7ea0 msgr2=0x7fbe6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.226+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 shutdown_connections 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.226+0000 7fbe7231c640 1 -- 172.21.15.176:0/536940700 wait complete. 2024-03-25T18:31:26.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:26.226+0000 7fbe7231c640 1 librados: shutdown 2024-03-25T18:31:26.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:33.046 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:33.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.161+0000 7f542cbd0640 10 monclient: get_monmap_and_config 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.161+0000 7f542cbd0640 10 monclient: build_initial_monmap 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.161+0000 7f542cbd0640 10 monclient: monmap: 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:33.162428+0000 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:33.162428+0000 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:33.163 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.162+0000 7f542cbd0640 1 Processor -- start 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.162+0000 7f542cbd0640 1 -- start start 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.162+0000 7f542cbd0640 10 monclient: init 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-c con 0x7f54280a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-b con 0x7f542813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-a con 0x7f54281430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5428142520 con 0x7f54281430c0 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280b28a0 con 0x7f54280a98f0 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280b2a30 con 0x7f542813f710 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:33.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.163+0000 7f542cbd0640 10 monclient(hunting): authenticate will time out at 8571.475557s 2024-03-25T18:31:33.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.167+0000 7f542659c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f54280a98f0 0x7f54280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:41830/0 (socket says 172.21.15.176:41830) 2024-03-25T18:31:33.168 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.167+0000 7f542659c640 1 -- 172.21.15.176:0/1940633791 learned_addr learned my addr 172.21.15.176:0/1940633791 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:33.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127145721 0 0) 0x7f54280b28a0 con 0x7f54280a98f0 2024-03-25T18:31:33.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:31:33.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.181+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5410003520 con 0x7f54280a98f0 2024-03-25T18:31:33.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527116692 0 0) 0x7f5428142520 con 0x7f54281430c0 2024-03-25T18:31:33.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:31:33.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.190+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54280b28a0 con 0x7f54281430c0 2024-03-25T18:31:33.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069722500 0 0) 0x7f54280b2a30 con 0x7f542813f710 2024-03-25T18:31:33.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:31:33.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.194+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428142520 con 0x7f542813f710 2024-03-25T18:31:33.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591376324 0 0) 0x7f5428142520 con 0x7f542813f710 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 >> v1:172.21.15.186:6789/0 conn(0x7f54280a98f0 legacy=0x7f54280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 >> v1:172.21.15.176:6789/0 conn(0x7f54281430c0 legacy=0x7f542814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: _finish_hunting 0 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 monclient: found mon.noname-b 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54280b44c0 con 0x7f542813f710 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: _finish_auth 0 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: _check_auth_tickets 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f541c0015b0 con 0x7f542813f710 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542559a640 10 monclient: dump: 2024-03-25T18:31:33.209 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542cbd0640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:31:33.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.207+0000 7f542cbd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f541c001710 con 0x7f542813f710 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 1 -- 172.21.15.176:0/1940633791 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f541c001b60 con 0x7f542813f710 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542559a640 10 monclient: dump: 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.225 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:33.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 4 set_mon_vals no callback set 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 10 monclient: shutdown 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 1 -- 172.21.15.176:0/1940633791 >> v1:172.21.15.176:6790/0 conn(0x7f542813f710 legacy=0x7f54281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 1 -- 172.21.15.176:0/1940633791 shutdown_connections 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 1 -- 172.21.15.176:0/1940633791 >> 172.21.15.176:0/1940633791 conn(0x7f54280b7ea0 msgr2=0x7f54280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:33.228 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 1 -- 172.21.15.176:0/1940633791 shutdown_connections 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.224+0000 7f542cbd0640 1 -- 172.21.15.176:0/1940633791 wait complete. 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.225+0000 7f542cbd0640 10 monclient: build_initial_monmap 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.225+0000 7f542cbd0640 10 monclient: monmap: 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 librados: starting msgr at 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 librados: starting objecter 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 Processor -- start 2024-03-25T18:31:33.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 -- start start 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 librados: setting wanted keys 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 librados: calling monclient init 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient: init 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-a con 0x7f54280a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-c con 0x7f542813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient(hunting): picked mon.noname-b con 0x7f54281430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542813b920 con 0x7f54280a98f0 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54281319b0 con 0x7f54281430c0 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542813b400 con 0x7f542813f710 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:33.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.226+0000 7f542cbd0640 10 monclient(hunting): authenticate will time out at 8571.538768s 2024-03-25T18:31:33.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.237+0000 7f5426d9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f54281430c0 0x7f54281312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:41884/0 (socket says 172.21.15.176:41884) 2024-03-25T18:31:33.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.237+0000 7f5426d9d640 1 -- 172.21.15.176:0/3482974790 learned_addr learned my addr 172.21.15.176:0/3482974790 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:33.239 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.238+0000 7f5416ffd640 10 client.?.objecter ms_handle_connect 0x7f54281430c0 2024-03-25T18:31:33.239 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.238+0000 7f5416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:33.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.245+0000 7f5416ffd640 10 client.?.objecter ms_handle_connect 0x7f54280a98f0 2024-03-25T18:31:33.246 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.245+0000 7f5416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:33.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.247+0000 7f5416ffd640 10 client.?.objecter ms_handle_connect 0x7f542813f710 2024-03-25T18:31:33.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.247+0000 7f5416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:33.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633537116 0 0) 0x7f54281319b0 con 0x7f54281430c0 2024-03-25T18:31:33.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:31:33.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.252+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408004eb0 con 0x7f54281430c0 2024-03-25T18:31:33.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700485063 0 0) 0x7f542813b920 con 0x7f54280a98f0 2024-03-25T18:31:33.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:31:33.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.260+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54281319b0 con 0x7f54280a98f0 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4099691147 0 0) 0x7f54281319b0 con 0x7f54280a98f0 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542813b920 con 0x7f54280a98f0 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5418001950 con 0x7f54280a98f0 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:33.280 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3864737955 0 0) 0x7f5408004eb0 con 0x7f54281430c0 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54281319b0 con 0x7f54281430c0 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5420001c90 con 0x7f54281430c0 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:33.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.278+0000 7f5416ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578182513 0 0) 0x7f542813b400 con 0x7f542813f710 2024-03-25T18:31:33.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:33.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:33.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:33.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:31:33.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.279+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408004eb0 con 0x7f542813f710 2024-03-25T18:31:33.292 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 267513259 0 0) 0x7f542813b920 con 0x7f54280a98f0 2024-03-25T18:31:33.292 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 >> v1:172.21.15.176:6790/0 conn(0x7f542813f710 legacy=0x7f5428130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 >> v1:172.21.15.186:6789/0 conn(0x7f54281430c0 legacy=0x7f54281312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 1 monclient: found mon.a 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428135070 con 0x7f54280a98f0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f5416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 10 monclient: _renew_subs 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5428135260 con 0x7f54280a98f0 2024-03-25T18:31:33.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 10 monclient: _renew_subs 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54280b7aa0 con 0x7f54280a98f0 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.291+0000 7f542cbd0640 1 librados: init done 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.292+0000 7f542cbd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:33.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.292+0000 7f542cbd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:33.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.292+0000 7f542cbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:33.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.292+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54280627b0 con 0x7f54280a98f0 2024-03-25T18:31:33.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5418001d60 con 0x7f54280a98f0 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f5418004980 con 0x7f54280a98f0 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5416ffd640 10 monclient: dump: 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:33.627 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:33.628 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:33.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.625+0000 7f5424d99640 4 set_mon_vals no callback set 2024-03-25T18:31:33.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.626+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f541801a990 con 0x7f54280a98f0 2024-03-25T18:31:33.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.626+0000 7f5425d9b640 10 monclient: get_auth_request con 0x7f540806df00 auth_method 0 2024-03-25T18:31:33.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.627+0000 7f5416ffd640 10 client.4539.objecter ms_handle_connect 0x7f540806df00 2024-03-25T18:31:33.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f5418054760 con 0x7f54280a98f0 2024-03-25T18:31:33.633 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 10 client.4539.objecter ms_dispatch 0x7f5428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54180500a0 con 0x7f54280a98f0 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:33.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.632+0000 7f5416ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:33.813 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:35.569 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.823 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:35.578 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.856+0000 7f542cbd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:33.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.856+0000 7f542cbd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:33.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.856+0000 7f542cbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:33.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.856+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f542813e2b0 con 0x7f54280a98f0 2024-03-25T18:31:33.881 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:35.615 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.885 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:35.641 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.890 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:35.624 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.891 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:35.647 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.895 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:35.650 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.917 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:35.651 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.918+0000 7f5416ffd640 1 -- 172.21.15.176:0/3482974790 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f54180521a0 con 0x7f54280a98f0 2024-03-25T18:31:33.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.918+0000 7f5416ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:33.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.918+0000 7f5416ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:33.920 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:33.920 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.920+0000 7f542cbd0640 10 librados: watch_flush enter 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.920+0000 7f542cbd0640 10 librados: watch_flush exit 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.920+0000 7f542cbd0640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.920+0000 7f542cbd0640 10 client.4539.objecter successfully canceled tick 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 >> v1:172.21.15.176:6800/2598213983 conn(0x7f540806df00 legacy=0x7f54080703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 10 monclient: shutdown 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 >> v1:172.21.15.176:6789/0 conn(0x7f54280a98f0 legacy=0x7f54281d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 shutdown_connections 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 >> 172.21.15.176:0/3482974790 conn(0x7f54280b7ea0 msgr2=0x7f5428074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 shutdown_connections 2024-03-25T18:31:33.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 -- 172.21.15.176:0/3482974790 wait complete. 2024-03-25T18:31:33.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:33.921+0000 7f542cbd0640 1 librados: shutdown 2024-03-25T18:31:33.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:33.935 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:35.691 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:40.739 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:40.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.857+0000 7fc9175a4640 10 monclient: get_monmap_and_config 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.857+0000 7fc9175a4640 10 monclient: build_initial_monmap 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.857+0000 7fc9175a4640 10 monclient: monmap: 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:40.858552+0000 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:40.858552+0000 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:40.859 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.858+0000 7fc9175a4640 1 Processor -- start 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.858+0000 7fc9175a4640 1 -- start start 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.858+0000 7fc9175a4640 10 monclient: init 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-a con 0x7fc9100a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-b con 0x7fc91013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9101430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc910142520 con 0x7fc9100a98f0 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100b28a0 con 0x7fc9101430c0 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100b2a30 con 0x7fc91013f710 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:40.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.859+0000 7fc9175a4640 10 monclient(hunting): authenticate will time out at 8579.171435s 2024-03-25T18:31:40.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.867+0000 7fc914b3f640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc91013f710 0x7fc9101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39190/0 (socket says 172.21.15.176:39190) 2024-03-25T18:31:40.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.867+0000 7fc914b3f640 1 -- 172.21.15.176:0/883064659 learned_addr learned my addr 172.21.15.176:0/883064659 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604341152 0 0) 0x7fc9100b2a30 con 0x7fc91013f710 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:31:40.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.883+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8fc003520 con 0x7fc91013f710 2024-03-25T18:31:40.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660938736 0 0) 0x7fc9100b28a0 con 0x7fc9101430c0 2024-03-25T18:31:40.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:31:40.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.885+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9100b2a30 con 0x7fc9101430c0 2024-03-25T18:31:40.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219803829 0 0) 0x7fc910142520 con 0x7fc9100a98f0 2024-03-25T18:31:40.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:31:40.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.890+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9100b28a0 con 0x7fc9100a98f0 2024-03-25T18:31:40.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446216697 0 0) 0x7fc8fc003520 con 0x7fc91013f710 2024-03-25T18:31:40.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 >> v1:172.21.15.186:6789/0 conn(0x7fc9101430c0 legacy=0x7fc91014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 >> v1:172.21.15.176:6789/0 conn(0x7fc9100a98f0 legacy=0x7fc9100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 monclient: found mon.noname-b 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100b44c0 con 0x7fc91013f710 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc900001690 con 0x7fc91013f710 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc907fff640 10 monclient: dump: 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:40.897 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc9175a4640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:31:40.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.895+0000 7fc9175a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9000017f0 con 0x7fc91013f710 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 1 -- 172.21.15.176:0/883064659 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc900001c00 con 0x7fc91013f710 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc9175a4640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc9175a4640 4 set_mon_vals no callback set 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.921+0000 7fc907fff640 10 monclient: dump: 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.923 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:40.924 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 10 monclient: shutdown 2024-03-25T18:31:40.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:40.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 1 -- 172.21.15.176:0/883064659 >> v1:172.21.15.176:6790/0 conn(0x7fc91013f710 legacy=0x7fc9101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:40.925 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 1 -- 172.21.15.176:0/883064659 shutdown_connections 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 1 -- 172.21.15.176:0/883064659 >> 172.21.15.176:0/883064659 conn(0x7fc9100b7ea0 msgr2=0x7fc9100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 1 -- 172.21.15.176:0/883064659 shutdown_connections 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.922+0000 7fc9175a4640 1 -- 172.21.15.176:0/883064659 wait complete. 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.923+0000 7fc9175a4640 10 monclient: build_initial_monmap 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.923+0000 7fc9175a4640 10 monclient: monmap: 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 librados: starting msgr at 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 librados: starting objecter 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 Processor -- start 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 -- start start 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 librados: setting wanted keys 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 1 librados: calling monclient init 2024-03-25T18:31:40.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.924+0000 7fc9175a4640 10 monclient: init 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-b con 0x7fc9100a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-c con 0x7fc91013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient(hunting): picked mon.noname-a con 0x7fc9101430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9101deec0 con 0x7fc9101430c0 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9101df1a0 con 0x7fc9100a98f0 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9101dfd40 con 0x7fc91013f710 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:40.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.925+0000 7fc9175a4640 10 monclient(hunting): authenticate will time out at 8579.237321s 2024-03-25T18:31:40.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.936+0000 7fc915b41640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc9101430c0 0x7fc91006ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:35130/0 (socket says 172.21.15.176:35130) 2024-03-25T18:31:40.938 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.936+0000 7fc915b41640 1 -- 172.21.15.176:0/772821298 learned_addr learned my addr 172.21.15.176:0/772821298 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:40.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.938+0000 7fc905ffb640 10 client.?.objecter ms_handle_connect 0x7fc91013f710 2024-03-25T18:31:40.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.938+0000 7fc905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:40.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.940+0000 7fc905ffb640 10 client.?.objecter ms_handle_connect 0x7fc9101430c0 2024-03-25T18:31:40.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.940+0000 7fc905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:40.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.943+0000 7fc905ffb640 10 client.?.objecter ms_handle_connect 0x7fc9100a98f0 2024-03-25T18:31:40.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.943+0000 7fc905ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.946+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572939170 0 0) 0x7fc9101df1a0 con 0x7fc9100a98f0 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:31:40.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4004eb0 con 0x7fc9100a98f0 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868040417 0 0) 0x7fc9101deec0 con 0x7fc9101430c0 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:31:40.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.947+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9101df1a0 con 0x7fc9101430c0 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841070629 0 0) 0x7fc9101df1a0 con 0x7fc9101430c0 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9101deec0 con 0x7fc9101430c0 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc908001890 con 0x7fc9101430c0 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:40.960 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743465696 0 0) 0x7fc8f4004eb0 con 0x7fc9100a98f0 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9101df1a0 con 0x7fc9100a98f0 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc90c001c70 con 0x7fc9100a98f0 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.958+0000 7fc905ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:40.961 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:40.962 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046889558 0 0) 0x7fc9101dfd40 con 0x7fc91013f710 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:31:40.966 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.964+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4004eb0 con 0x7fc91013f710 2024-03-25T18:31:40.973 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291596852 0 0) 0x7fc9101deec0 con 0x7fc9101430c0 2024-03-25T18:31:40.973 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 >> v1:172.21.15.176:6790/0 conn(0x7fc91013f710 legacy=0x7fc9100716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 >> v1:172.21.15.186:6789/0 conn(0x7fc9100a98f0 legacy=0x7fc9101df630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 1 monclient: found mon.a 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91006e100 con 0x7fc9101430c0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 10 monclient: _renew_subs 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc910071dd0 con 0x7fc9101430c0 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:40.974 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 10 monclient: _renew_subs 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91006f5e0 con 0x7fc9101430c0 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.972+0000 7fc9175a4640 1 librados: init done 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.973+0000 7fc9175a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.973+0000 7fc9175a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.973+0000 7fc9175a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:40.975 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:40.973+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc910072120 con 0x7fc9101430c0 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc908003610 con 0x7fc9101430c0 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc9077fe640 4 set_mon_vals no callback set 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc908001d80 con 0x7fc9101430c0 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.291+0000 7fc905ffb640 10 monclient: dump: 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:41.293 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:41.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.292+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fc908003aa0 con 0x7fc9101430c0 2024-03-25T18:31:41.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.292+0000 7fc915340640 10 monclient: get_auth_request con 0x7fc8f406df50 auth_method 0 2024-03-25T18:31:41.295 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.292+0000 7fc905ffb640 10 client.4545.objecter ms_handle_connect 0x7fc8f406df50 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fc9080546e0 con 0x7fc9101430c0 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 10 client.4545.objecter ms_dispatch 0x7fc910000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9080501f0 con 0x7fc9101430c0 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:41.300 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.298+0000 7fc905ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.507+0000 7fc9175a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.507+0000 7fc9175a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.507+0000 7fc9175a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:31:41.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.507+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc91014f330 con 0x7fc9101430c0 2024-03-25T18:31:41.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.543+0000 7fc905ffb640 1 -- 172.21.15.176:0/772821298 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fc91014f330 con 0x7fc9101430c0 2024-03-25T18:31:41.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.543+0000 7fc905ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:41.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.543+0000 7fc905ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:41.545 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:41.545 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 10 librados: watch_flush enter 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 10 librados: watch_flush exit 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 >> v1:172.21.15.176:6800/2598213983 conn(0x7fc8f406df50 legacy=0x7fc8f4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 10 monclient: shutdown 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 >> v1:172.21.15.176:6789/0 conn(0x7fc9101430c0 legacy=0x7fc91006ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 shutdown_connections 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 >> 172.21.15.176:0/772821298 conn(0x7fc9100b7ea0 msgr2=0x7fc910074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 shutdown_connections 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 -- 172.21.15.176:0/772821298 wait complete. 2024-03-25T18:31:41.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:41.546+0000 7fc9175a4640 1 librados: shutdown 2024-03-25T18:31:41.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:48.365 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:48.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.482+0000 7fede5006640 10 monclient: get_monmap_and_config 2024-03-25T18:31:48.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.482+0000 7fede5006640 10 monclient: build_initial_monmap 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.482+0000 7fede5006640 10 monclient: monmap: 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:48.483142+0000 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:48.483142+0000 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:48.484 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.482+0000 7fede5006640 1 Processor -- start 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 1 -- start start 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient: init 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient(hunting): picked mon.noname-c con 0x7fede00a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient(hunting): picked mon.noname-a con 0x7fede013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient(hunting): picked mon.noname-b con 0x7fede01430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede0142520 con 0x7fede013f710 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00b28a0 con 0x7fede00a98f0 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede00b2a30 con 0x7fede01430c0 2024-03-25T18:31:48.486 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:48.487 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.483+0000 7fede5006640 10 monclient(hunting): authenticate will time out at 8586.795747s 2024-03-25T18:31:48.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.498+0000 7feddf59e640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fede01430c0 0x7fede014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:56280/0 (socket says 172.21.15.176:56280) 2024-03-25T18:31:48.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.498+0000 7feddf59e640 1 -- 172.21.15.176:0/1033261359 learned_addr learned my addr 172.21.15.176:0/1033261359 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160764510 0 0) 0x7fede00b2a30 con 0x7fede01430c0 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:31:48.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.510+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8003520 con 0x7fede01430c0 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262077214 0 0) 0x7fede0142520 con 0x7fede013f710 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:31:48.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede00b2a30 con 0x7fede013f710 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391038876 0 0) 0x7fede00b28a0 con 0x7fede00a98f0 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:31:48.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.515+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede0142520 con 0x7fede00a98f0 2024-03-25T18:31:48.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458103078 0 0) 0x7fedc8003520 con 0x7fede01430c0 2024-03-25T18:31:48.527 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 >> v1:172.21.15.186:6789/0 conn(0x7fede00a98f0 legacy=0x7fede00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 >> v1:172.21.15.176:6789/0 conn(0x7fede013f710 legacy=0x7fede01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 monclient: found mon.noname-b 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00b44c0 con 0x7fede01430c0 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fedd8001a10 con 0x7fede01430c0 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fedddd9b640 10 monclient: dump: 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:48.528 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fede5006640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:31:48.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.526+0000 7fede5006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:48.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedd8001b70 con 0x7fede01430c0 2024-03-25T18:31:48.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:48.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 1 -- 172.21.15.176:0/1033261359 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fedd8001fc0 con 0x7fede01430c0 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fedddd9b640 10 monclient: dump: 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fede5006640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:48.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.531+0000 7fede5006640 4 set_mon_vals no callback set 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 10 monclient: shutdown 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 1 -- 172.21.15.176:0/1033261359 >> v1:172.21.15.176:6790/0 conn(0x7fede01430c0 legacy=0x7fede014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 1 -- 172.21.15.176:0/1033261359 shutdown_connections 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 1 -- 172.21.15.176:0/1033261359 >> 172.21.15.176:0/1033261359 conn(0x7fede00b7ea0 msgr2=0x7fede00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 1 -- 172.21.15.176:0/1033261359 shutdown_connections 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.532+0000 7fede5006640 1 -- 172.21.15.176:0/1033261359 wait complete. 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 10 monclient: build_initial_monmap 2024-03-25T18:31:48.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 10 monclient: monmap: 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 librados: starting msgr at 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 librados: starting objecter 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 Processor -- start 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 -- start start 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 librados: setting wanted keys 2024-03-25T18:31:48.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 1 librados: calling monclient init 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.533+0000 7fede5006640 10 monclient: init 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.534+0000 7fede5006640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.534+0000 7fede5006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.534+0000 7fede5006640 10 monclient(hunting): picked mon.noname-a con 0x7fede00a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 10 monclient(hunting): picked mon.noname-b con 0x7fede013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 10 monclient(hunting): picked mon.noname-c con 0x7fede01430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:48.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede013c960 con 0x7fede00a98f0 2024-03-25T18:31:48.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede012f560 con 0x7fede013f710 2024-03-25T18:31:48.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede012f840 con 0x7fede01430c0 2024-03-25T18:31:48.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:48.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.535+0000 7fede5006640 10 monclient(hunting): authenticate will time out at 8586.846896s 2024-03-25T18:31:48.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.546+0000 7fedde59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fede013f710 0x7fede0130650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:50386/0 (socket says 172.21.15.176:50386) 2024-03-25T18:31:48.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.546+0000 7fedde59c640 1 -- 172.21.15.176:0/3366529056 learned_addr learned my addr 172.21.15.176:0/3366529056 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:48.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.549+0000 7fedc77fe640 10 client.?.objecter ms_handle_connect 0x7fede013f710 2024-03-25T18:31:48.550 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.549+0000 7fedc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:48.553 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.552+0000 7fedc77fe640 10 client.?.objecter ms_handle_connect 0x7fede00a98f0 2024-03-25T18:31:48.553 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.552+0000 7fedc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:48.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.554+0000 7fedc77fe640 10 client.?.objecter ms_handle_connect 0x7fede01430c0 2024-03-25T18:31:48.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.554+0000 7fedc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:48.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355798132 0 0) 0x7fede012f560 con 0x7fede013f710 2024-03-25T18:31:48.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:31:48.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.558+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedcc004eb0 con 0x7fede013f710 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297838881 0 0) 0x7fede013c960 con 0x7fede00a98f0 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:31:48.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.563+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede012f560 con 0x7fede00a98f0 2024-03-25T18:31:48.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203001835 0 0) 0x7fedcc004eb0 con 0x7fede013f710 2024-03-25T18:31:48.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fede013c960 con 0x7fede013f710 2024-03-25T18:31:48.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fedd4001a30 con 0x7fede013f710 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 10 monclient(hunting): dump: 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.567+0000 7fedc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:48.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008746680 0 0) 0x7fede012f840 con 0x7fede01430c0 2024-03-25T18:31:48.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:48.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:48.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:48.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:31:48.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.571+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedcc004eb0 con 0x7fede01430c0 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 553507499 0 0) 0x7fede013c960 con 0x7fede013f710 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 >> v1:172.21.15.176:6790/0 conn(0x7fede01430c0 legacy=0x7fede0130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 >> v1:172.21.15.176:6789/0 conn(0x7fede00a98f0 legacy=0x7fede012ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 1 monclient: found mon.b 2024-03-25T18:31:48.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede012fb20 con 0x7fede013f710 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fedc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 10 monclient: _renew_subs 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede013bfb0 con 0x7fede013f710 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:48.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 10 monclient: _renew_subs 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fede01398c0 con 0x7fede013f710 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.573+0000 7fede5006640 1 librados: init done 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.574+0000 7fede5006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.574+0000 7fede5006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.574+0000 7fede5006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:48.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.574+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede00627b0 con 0x7fede013f710 2024-03-25T18:31:48.853 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedd4001c70 con 0x7fede013f710 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fedd4003a60 con 0x7fede013f710 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7fedc77fe640 10 monclient: dump: 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:48.854 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.852+0000 7feddd59a640 4 set_mon_vals no callback set 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.853+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fedd4003c80 con 0x7fede013f710 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.853+0000 7fedded9d640 10 monclient: get_auth_request con 0x7fedcc06de00 auth_method 0 2024-03-25T18:31:48.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.854+0000 7fedc77fe640 10 client.4564.objecter ms_handle_connect 0x7fedcc06de00 2024-03-25T18:31:48.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fedd4054860 con 0x7fede013f710 2024-03-25T18:31:48.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 10 client.4564.objecter ms_dispatch 0x7fede0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:48.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:48.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:48.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:31:48.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedd4050370 con 0x7fede013f710 2024-03-25T18:31:48.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:48.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:48.859+0000 7fedc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:49.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.112+0000 7fede5006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:49.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.112+0000 7fede5006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:49.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.112+0000 7fede5006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:49.114 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.112+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede014e780 con 0x7fede013f710 2024-03-25T18:31:49.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.142+0000 7fedc77fe640 1 -- 172.21.15.176:0/3366529056 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fedd40522a0 con 0x7fede013f710 2024-03-25T18:31:49.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.142+0000 7fedc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:49.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.142+0000 7fedc77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:49.144 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:49.144 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 10 librados: watch_flush enter 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 10 librados: watch_flush exit 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 >> v1:172.21.15.176:6800/2598213983 conn(0x7fedcc06de00 legacy=0x7fedcc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 10 monclient: shutdown 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 >> v1:172.21.15.186:6789/0 conn(0x7fede013f710 legacy=0x7fede0130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 shutdown_connections 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 >> 172.21.15.176:0/3366529056 conn(0x7fede00b7ea0 msgr2=0x7fede0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 shutdown_connections 2024-03-25T18:31:49.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 -- 172.21.15.176:0/3366529056 wait complete. 2024-03-25T18:31:49.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:49.145+0000 7fede5006640 1 librados: shutdown 2024-03-25T18:31:49.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:53.856 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:55.612 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.869 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:55.625 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.907 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:07:55.663 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.911 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:55.646 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.920 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:55.654 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.929 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:55.685 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.946 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:55.702 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.955 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:55.689 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.970 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:07:55.726 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.966 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.078+0000 7fe1cbd0e640 10 monclient: get_monmap_and_config 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.079+0000 7fe1cbd0e640 10 monclient: build_initial_monmap 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.079+0000 7fe1cbd0e640 10 monclient: monmap: 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:31:56.079877+0000 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:31:56.079877+0000 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:56.080 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:31:56.081 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:31:56.081 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.079+0000 7fe1cbd0e640 1 Processor -- start 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 1 -- start start 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient: init 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c40a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe1c41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c4142520 con 0x7fe1c40a98f0 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40b28a0 con 0x7fe1c41430c0 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40b2a30 con 0x7fe1c413f710 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:56.082 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.080+0000 7fe1cbd0e640 10 monclient(hunting): authenticate will time out at 8594.392515s 2024-03-25T18:31:56.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.086+0000 7fe1ca2ab640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe1c41430c0 0x7fe1c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:35002/0 (socket says 172.21.15.176:35002) 2024-03-25T18:31:56.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.086+0000 7fe1ca2ab640 1 -- 172.21.15.176:0/689028919 learned_addr learned my addr 172.21.15.176:0/689028919 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:56.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486355937 0 0) 0x7fe1c40b28a0 con 0x7fe1c41430c0 2024-03-25T18:31:56.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:56.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:56.096 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:56.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:31:56.097 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.095+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b0003520 con 0x7fe1c41430c0 2024-03-25T18:31:56.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495680375 0 0) 0x7fe1b0003520 con 0x7fe1c41430c0 2024-03-25T18:31:56.106 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 >> v1:172.21.15.176:6790/0 conn(0x7fe1c413f710 legacy=0x7fe1c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 >> v1:172.21.15.176:6789/0 conn(0x7fe1c40a98f0 legacy=0x7fe1c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: _finish_hunting 0 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 monclient: found mon.noname-c 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40b44c0 con 0x7fe1c41430c0 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: _finish_auth 0 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: _check_auth_tickets 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe1bc001a10 con 0x7fe1c41430c0 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1c8aa8640 10 monclient: dump: 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:56.107 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1cbd0e640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:31:56.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.105+0000 7fe1cbd0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:56.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1bc001b70 con 0x7fe1c41430c0 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 1 -- 172.21.15.176:0/689028919 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe1bc001f80 con 0x7fe1c41430c0 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1cbd0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1cbd0e640 4 set_mon_vals no callback set 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.111+0000 7fe1c8aa8640 10 monclient: dump: 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:56.113 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 10 monclient: shutdown 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/689028919 >> v1:172.21.15.186:6789/0 conn(0x7fe1c41430c0 legacy=0x7fe1c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/689028919 shutdown_connections 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/689028919 >> 172.21.15.176:0/689028919 conn(0x7fe1c40b7ea0 msgr2=0x7fe1c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.112+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/689028919 shutdown_connections 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.113+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/689028919 wait complete. 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.113+0000 7fe1cbd0e640 10 monclient: build_initial_monmap 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.113+0000 7fe1cbd0e640 10 monclient: monmap: 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:56.115 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 librados: starting msgr at 2024-03-25T18:31:56.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 librados: starting objecter 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 Processor -- start 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 -- start start 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 librados: setting wanted keys 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 1 librados: calling monclient init 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.114+0000 7fe1cbd0e640 10 monclient: init 2024-03-25T18:31:56.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe1c40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40abc00 con 0x7fe1c41430c0 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40ac150 con 0x7fe1c413f710 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c40abe70 con 0x7fe1c40a98f0 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:56.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.115+0000 7fe1cbd0e640 10 monclient(hunting): authenticate will time out at 8594.427405s 2024-03-25T18:31:56.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.133+0000 7fe1ca2ab640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe1c41430c0 0x7fe1c41e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:52650/0 (socket says 172.21.15.176:52650) 2024-03-25T18:31:56.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.133+0000 7fe1ca2ab640 1 -- 172.21.15.176:0/1785589300 learned_addr learned my addr 172.21.15.176:0/1785589300 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:31:56.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.134+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1c41430c0 2024-03-25T18:31:56.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.134+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:56.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.134+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1c413f710 2024-03-25T18:31:56.135 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.134+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:56.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.140+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1c40a98f0 2024-03-25T18:31:56.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.140+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.141+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856253672 0 0) 0x7fe1c40ac150 con 0x7fe1c413f710 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.142+0000 7fe1ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.142+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.142+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.142+0000 7fe1ba7fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:31:56.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.142+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a8004eb0 con 0x7fe1c413f710 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393719008 0 0) 0x7fe1c40abe70 con 0x7fe1c40a98f0 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:31:56.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.146+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c40ac150 con 0x7fe1c40a98f0 2024-03-25T18:31:56.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910664024 0 0) 0x7fe1c40abc00 con 0x7fe1c41430c0 2024-03-25T18:31:56.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:56.150 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:56.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:56.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:31:56.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.149+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c40abe70 con 0x7fe1c41430c0 2024-03-25T18:31:56.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218336147 0 0) 0x7fe1a8004eb0 con 0x7fe1c413f710 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c40abc00 con 0x7fe1c413f710 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe1b40018c0 con 0x7fe1c413f710 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 10 monclient(hunting): dump: 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.153+0000 7fe1ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 433966285 0 0) 0x7fe1c40abc00 con 0x7fe1c413f710 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 >> v1:172.21.15.176:6790/0 conn(0x7fe1c40a98f0 legacy=0x7fe1c4136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 >> v1:172.21.15.176:6789/0 conn(0x7fe1c41430c0 legacy=0x7fe1c41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 1 monclient: found mon.b 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c41390f0 con 0x7fe1c413f710 2024-03-25T18:31:56.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1cbd0e640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1cbd0e640 10 monclient: _renew_subs 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1cbd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.177+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c4137e00 con 0x7fe1c413f710 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.178+0000 7fe1cbd0e640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.178+0000 7fe1cbd0e640 10 monclient: _renew_subs 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.178+0000 7fe1cbd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.178+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c4138bf0 con 0x7fe1c413f710 2024-03-25T18:31:56.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.178+0000 7fe1cbd0e640 1 librados: init done 2024-03-25T18:31:56.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.179+0000 7fe1cbd0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:56.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.179+0000 7fe1cbd0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:56.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.179+0000 7fe1cbd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.179+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c4138ef0 con 0x7fe1c413f710 2024-03-25T18:31:56.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.183+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1b40035f0 con 0x7fe1c413f710 2024-03-25T18:31:56.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.183+0000 7fe1ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:56.184 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.183+0000 7fe1bbfff640 4 set_mon_vals no callback set 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.454+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe1b4003a00 con 0x7fe1c413f710 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.454+0000 7fe1ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.454+0000 7fe1ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.454+0000 7fe1ba7fc640 10 monclient: dump: 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:31:56.456 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.454+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fe1b4003be0 con 0x7fe1c413f710 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.455+0000 7fe1c9aaa640 10 monclient: get_auth_request con 0x7fe1a806dda0 auth_method 0 2024-03-25T18:31:56.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.455+0000 7fe1ba7fc640 10 client.4570.objecter ms_handle_connect 0x7fe1a806dda0 2024-03-25T18:31:56.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.460+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fe1b4053620 con 0x7fe1c413f710 2024-03-25T18:31:56.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.460+0000 7fe1ba7fc640 10 client.4570.objecter ms_dispatch 0x7fe1c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:56.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.460+0000 7fe1ba7fc640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:56.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.460+0000 7fe1ba7fc640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:56.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.460+0000 7fe1ba7fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:31:56.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.464+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1b404f130 con 0x7fe1c413f710 2024-03-25T18:31:56.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.464+0000 7fe1ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:56.465 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.464+0000 7fe1ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:56.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.684+0000 7fe1cbd0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:56.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.684+0000 7fe1cbd0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:56.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.684+0000 7fe1cbd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:31:56.686 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.684+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1c40627b0 con 0x7fe1c413f710 2024-03-25T18:31:56.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.704+0000 7fe1ba7fc640 1 -- 172.21.15.176:0/1785589300 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fe1b4051060 con 0x7fe1c413f710 2024-03-25T18:31:56.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.705+0000 7fe1ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:56.706 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.705+0000 7fe1ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:56.706 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:31:56.706 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 10 librados: watch_flush enter 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 10 librados: watch_flush exit 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 >> v1:172.21.15.176:6800/2598213983 conn(0x7fe1a806dda0 legacy=0x7fe1a8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 10 monclient: shutdown 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 >> v1:172.21.15.186:6789/0 conn(0x7fe1c413f710 legacy=0x7fe1c40ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 shutdown_connections 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 >> 172.21.15.176:0/1785589300 conn(0x7fe1c40b7ea0 msgr2=0x7fe1c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 shutdown_connections 2024-03-25T18:31:56.709 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 -- 172.21.15.176:0/1785589300 wait complete. 2024-03-25T18:31:56.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:31:56.707+0000 7fe1cbd0e640 1 librados: shutdown 2024-03-25T18:31:56.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:03.527 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 10 monclient: get_monmap_and_config 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 10 monclient: build_initial_monmap 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 10 monclient: monmap: 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:03.641923+0000 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:03.641923+0000 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:03.642 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:03.643 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 1 Processor -- start 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 1 -- start start 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.641+0000 7f3c22cae640 10 monclient: init 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-a con 0x7f3c1c0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0af940 con 0x7f3c1c0b1730 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0b0a70 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0b0d50 con 0x7f3c1c14be90 2024-03-25T18:32:03.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:03.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.642+0000 7f3c22cae640 10 monclient(hunting): authenticate will time out at 8601.954707s 2024-03-25T18:32:03.653 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.652+0000 7f3c2124b640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f3c1c14be90 0x7f3c1c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:55274/0 (socket says 172.21.15.176:55274) 2024-03-25T18:32:03.654 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.652+0000 7f3c2124b640 1 -- 172.21.15.176:0/245727613 learned_addr learned my addr 172.21.15.176:0/245727613 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.672+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332699365 0 0) 0x7f3c1c0b0a70 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.673+0000 7f3c1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.673+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.673+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.673+0000 7f3c1b7fe640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:32:03.674 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.673+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c003520 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894338134 0 0) 0x7f3c1c0b0d50 con 0x7f3c1c14be90 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:32:03.680 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.679+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c0b0a70 con 0x7f3c1c14be90 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2962717864 0 0) 0x7f3c1c0b0a70 con 0x7f3c1c14be90 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 >> v1:172.21.15.186:6789/0 conn(0x7f3c1c0b4d60 legacy=0x7f3c1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 >> v1:172.21.15.176:6789/0 conn(0x7f3c1c0b1730 legacy=0x7f3c1c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:32:03.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0b1030 con 0x7f3c1c14be90 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3c140019f0 con 0x7f3c1c14be90 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c22cae640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c22cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c1b7fe640 10 monclient: dump: 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:03.698 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.699 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.696+0000 7f3c22cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:03.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c14001b50 con 0x7f3c1c14be90 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 1 -- 172.21.15.176:0/245727613 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3c14001f20 con 0x7f3c1c14be90 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c22cae640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c22cae640 4 set_mon_vals no callback set 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.711+0000 7f3c1b7fe640 10 monclient: dump: 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:03.713 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:03.714 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 10 monclient: shutdown 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 1 -- 172.21.15.176:0/245727613 >> v1:172.21.15.176:6790/0 conn(0x7f3c1c14be90 legacy=0x7f3c1c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 1 -- 172.21.15.176:0/245727613 shutdown_connections 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 1 -- 172.21.15.176:0/245727613 >> 172.21.15.176:0/245727613 conn(0x7f3c1c0617a0 msgr2=0x7f3c1c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 1 -- 172.21.15.176:0/245727613 shutdown_connections 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.712+0000 7f3c22cae640 1 -- 172.21.15.176:0/245727613 wait complete. 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.713+0000 7f3c22cae640 10 monclient: build_initial_monmap 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.713+0000 7f3c22cae640 10 monclient: monmap: 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:03.716 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.713+0000 7f3c22cae640 1 librados: starting msgr at 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.713+0000 7f3c22cae640 1 librados: starting objecter 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.713+0000 7f3c22cae640 1 Processor -- start 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 -- start start 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 librados: setting wanted keys 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 librados: calling monclient init 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient: init 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-a con 0x7f3c1c0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:03.717 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c142080 con 0x7f3c1c0b1730 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c140dc0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c1408a0 con 0x7f3c1c14be90 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:03.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.714+0000 7f3c22cae640 10 monclient(hunting): authenticate will time out at 8602.026726s 2024-03-25T18:32:03.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.722+0000 7f3c1bfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3c1c0b4d60 0x7f3c1c131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:39852/0 (socket says 172.21.15.176:39852) 2024-03-25T18:32:03.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.722+0000 7f3c1bfff640 1 -- 172.21.15.176:0/1319858329 learned_addr learned my addr 172.21.15.176:0/1319858329 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:03.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.726+0000 7f3c197fa640 10 client.?.objecter ms_handle_connect 0x7f3c1c0b4d60 2024-03-25T18:32:03.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.726+0000 7f3c197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:03.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.732+0000 7f3c197fa640 10 client.?.objecter ms_handle_connect 0x7f3c1c0b1730 2024-03-25T18:32:03.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.732+0000 7f3c197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821407821 0 0) 0x7f3c1c140dc0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.734+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c00004eb0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.735+0000 7f3c197fa640 10 client.?.objecter ms_handle_connect 0x7f3c1c14be90 2024-03-25T18:32:03.736 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.735+0000 7f3c197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:03.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182147978 0 0) 0x7f3c1c142080 con 0x7f3c1c0b1730 2024-03-25T18:32:03.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:03.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:03.740 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:03.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:32:03.741 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.739+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c140dc0 con 0x7f3c1c0b1730 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840427072 0 0) 0x7f3c1c1408a0 con 0x7f3c1c14be90 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:32:03.743 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.742+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c142080 con 0x7f3c1c14be90 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 811404537 0 0) 0x7f3c00004eb0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c1c1408a0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3c08001a30 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 10 monclient(hunting): dump: 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:03.748 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.747+0000 7f3c197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:03.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.751+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1535997379 0 0) 0x7f3c1c142080 con 0x7f3c1c14be90 2024-03-25T18:32:03.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.751+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c00004eb0 con 0x7f3c1c14be90 2024-03-25T18:32:03.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.752+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3c14001e60 con 0x7f3c1c14be90 2024-03-25T18:32:03.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.752+0000 7f3c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:03.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.752+0000 7f3c197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.752+0000 7f3c197fa640 10 monclient(hunting): dump: 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:03.754 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1756226511 0 0) 0x7f3c1c1408a0 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 >> v1:172.21.15.176:6790/0 conn(0x7f3c1c14be90 legacy=0x7f3c1c1e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 >> v1:172.21.15.176:6789/0 conn(0x7f3c1c0b1730 legacy=0x7f3c1c1dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 1 monclient: found mon.b 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c137f40 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 10 monclient: _renew_subs 2024-03-25T18:32:03.756 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c138790 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 10 monclient: _renew_subs 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c140560 con 0x7f3c1c0b4d60 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.753+0000 7f3c22cae640 1 librados: init done 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.754+0000 7f3c22cae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.754+0000 7f3c22cae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.754+0000 7f3c22cae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:03.757 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:03.754+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c0627b0 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c08001c70 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3c08003a60 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c197fa640 10 monclient: dump: 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:04.025 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:04.026 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.023+0000 7f3c1affd640 4 set_mon_vals no callback set 2024-03-25T18:32:04.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.025+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f3c08003c40 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.025+0000 7f3c20a4a640 10 monclient: get_auth_request con 0x7f3c0006def0 auth_method 0 2024-03-25T18:32:04.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.025+0000 7f3c197fa640 10 client.4576.objecter ms_handle_connect 0x7f3c0006def0 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f3c080548b0 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 10 client.4576.objecter ms_dispatch 0x7f3c1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:32:04.031 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c080503c0 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:04.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.030+0000 7f3c197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:04.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.259+0000 7f3c22cae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:04.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.259+0000 7f3c22cae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:04.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.259+0000 7f3c22cae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:04.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.259+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c1c069f40 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.290 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.289+0000 7f3c197fa640 1 -- 172.21.15.176:0/1319858329 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f3c080522f0 con 0x7f3c1c0b4d60 2024-03-25T18:32:04.290 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.289+0000 7f3c197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:04.290 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.289+0000 7f3c197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:04.290 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:04.290 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:04.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.291+0000 7f3c22cae640 10 librados: watch_flush enter 2024-03-25T18:32:04.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 10 librados: watch_flush exit 2024-03-25T18:32:04.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:32:04.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 >> v1:172.21.15.176:6800/2598213983 conn(0x7f3c0006def0 legacy=0x7f3c000703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 10 monclient: shutdown 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 >> v1:172.21.15.186:6789/0 conn(0x7f3c1c0b4d60 legacy=0x7f3c1c131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 shutdown_connections 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 >> 172.21.15.176:0/1319858329 conn(0x7f3c1c0617a0 msgr2=0x7f3c1c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 shutdown_connections 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 -- 172.21.15.176:0/1319858329 wait complete. 2024-03-25T18:32:04.294 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:04.292+0000 7f3c22cae640 1 librados: shutdown 2024-03-25T18:32:04.304 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:11.110 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.226+0000 7f8fa2fc8640 10 monclient: get_monmap_and_config 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.226+0000 7f8fa2fc8640 10 monclient: build_initial_monmap 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.226+0000 7f8fa2fc8640 10 monclient: monmap: 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:11.226992+0000 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:11.226992+0000 2024-03-25T18:32:11.227 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:11.228 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.228 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:11.228 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:11.228 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:11.228 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.226+0000 7f8fa2fc8640 1 Processor -- start 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 1 -- start start 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient: init 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c13f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9c1430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c142520 con 0x7f8f9c13f710 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b28a0 con 0x7f8f9c1430c0 2024-03-25T18:32:11.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b2a30 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:11.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.227+0000 7f8fa2fc8640 10 monclient(hunting): authenticate will time out at 8609.539631s 2024-03-25T18:32:11.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.230+0000 7f8fa0d64640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f8f9c0a98f0 0x7f8f9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39438/0 (socket says 172.21.15.176:39438) 2024-03-25T18:32:11.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.230+0000 7f8fa0d64640 1 -- 172.21.15.176:0/197441890 learned_addr learned my addr 172.21.15.176:0/197441890 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115911308 0 0) 0x7f8f9c142520 con 0x7f8f9c13f710 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:32:11.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.239+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8f9c13f710 2024-03-25T18:32:11.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389231322 0 0) 0x7f8f9c0b2a30 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:11.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:11.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:11.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:32:11.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.241+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c142520 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162922239 0 0) 0x7f8f8c003520 con 0x7f8f9c13f710 2024-03-25T18:32:11.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 >> v1:172.21.15.176:6790/0 conn(0x7f8f9c0a98f0 legacy=0x7f8f9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 >> v1:172.21.15.186:6789/0 conn(0x7f8f9c1430c0 legacy=0x7f8f9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c0b44c0 con 0x7f8f9c13f710 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8f88001690 con 0x7f8f9c13f710 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8f9b7fe640 10 monclient: dump: 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:11.249 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8fa2fc8640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:32:11.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.247+0000 7f8fa2fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:11.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f88001870 con 0x7f8f9c13f710 2024-03-25T18:32:11.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:11.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 1 -- 172.21.15.176:0/197441890 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8f88001c80 con 0x7f8f9c13f710 2024-03-25T18:32:11.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8fa2fc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8fa2fc8640 4 set_mon_vals no callback set 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.250+0000 7f8f9b7fe640 10 monclient: dump: 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:11.252 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 10 monclient: shutdown 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/197441890 >> v1:172.21.15.176:6789/0 conn(0x7f8f9c13f710 legacy=0x7f8f9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/197441890 shutdown_connections 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/197441890 >> 172.21.15.176:0/197441890 conn(0x7f8f9c0b7ea0 msgr2=0x7f8f9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/197441890 shutdown_connections 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.251+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/197441890 wait complete. 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.252+0000 7f8fa2fc8640 10 monclient: build_initial_monmap 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.252+0000 7f8fa2fc8640 10 monclient: monmap: 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:11.254 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.252+0000 7f8fa2fc8640 1 librados: starting msgr at 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.252+0000 7f8fa2fc8640 1 librados: starting objecter 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.252+0000 7f8fa2fc8640 1 Processor -- start 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 -- start start 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 librados: setting wanted keys 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 librados: calling monclient init 2024-03-25T18:32:11.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient: init 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9c1430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c137cf0 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c1398d0 con 0x7f8f9c13f710 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0abb00 con 0x7f8f9c1430c0 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:11.257 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.253+0000 7f8fa2fc8640 10 monclient(hunting): authenticate will time out at 8609.565759s 2024-03-25T18:32:11.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.260+0000 7f8fa1565640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f8f9c1430c0 0x7f8f9c1e2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39464/0 (socket says 172.21.15.176:39464) 2024-03-25T18:32:11.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.260+0000 7f8fa1565640 1 -- 172.21.15.176:0/2903812553 learned_addr learned my addr 172.21.15.176:0/2903812553 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:11.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.260+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c1430c0 2024-03-25T18:32:11.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.260+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:11.269 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539259154 0 0) 0x7f8f9c0abb00 con 0x7f8f9c1430c0 2024-03-25T18:32:11.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:11.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:11.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:11.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:32:11.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.268+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80004eb0 con 0x7f8f9c1430c0 2024-03-25T18:32:11.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.271+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c13f710 2024-03-25T18:32:11.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.271+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:11.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.278+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c0a98f0 2024-03-25T18:32:11.279 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.278+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132640978 0 0) 0x7f8f80004eb0 con 0x7f8f9c1430c0 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9c0abb00 con 0x7f8f9c1430c0 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8f94001c90 con 0x7f8f9c1430c0 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 10 monclient(hunting): dump: 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.281 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.280+0000 7f8f997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242701168 0 0) 0x7f8f9c1398d0 con 0x7f8f9c13f710 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:32:11.288 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.286+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80004eb0 con 0x7f8f9c13f710 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.304+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466258992 0 0) 0x7f8f9c137cf0 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.305+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.305+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.305+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.305+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:32:11.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.305+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c1398d0 con 0x7f8f9c0a98f0 2024-03-25T18:32:11.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683622969 0 0) 0x7f8f80004eb0 con 0x7f8f9c13f710 2024-03-25T18:32:11.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9c137cf0 con 0x7f8f9c13f710 2024-03-25T18:32:11.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8f88001b80 con 0x7f8f9c13f710 2024-03-25T18:32:11.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:11.307 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.306+0000 7f8f997fa640 10 monclient(hunting): dump: 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:11.308 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 206692571 0 0) 0x7f8f9c0abb00 con 0x7f8f9c1430c0 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 >> v1:172.21.15.186:6789/0 conn(0x7f8f9c13f710 legacy=0x7f8f9c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 >> v1:172.21.15.176:6789/0 conn(0x7f8f9c0a98f0 legacy=0x7f8f9c1d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 1 monclient: found mon.c 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:11.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c137830 con 0x7f8f9c1430c0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8f997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 10 monclient: _renew_subs 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c138c70 con 0x7f8f9c1430c0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 10 monclient: _renew_subs 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c134e70 con 0x7f8f9c1430c0 2024-03-25T18:32:11.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.309+0000 7f8fa2fc8640 1 librados: init done 2024-03-25T18:32:11.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.310+0000 7f8fa2fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:11.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.310+0000 7f8fa2fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:11.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.310+0000 7f8fa2fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:11.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.310+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c135120 con 0x7f8f9c1430c0 2024-03-25T18:32:11.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f94003990 con 0x7f8f9c1430c0 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f8f94003b80 con 0x7f8f9c1430c0 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f997fa640 10 monclient: dump: 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:11.598 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.596+0000 7f8f9affd640 4 set_mon_vals no callback set 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.597+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f8f94003da0 con 0x7f8f9c1430c0 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.597+0000 7f8fa0d64640 10 monclient: get_auth_request con 0x7f8f8006dfd0 auth_method 0 2024-03-25T18:32:11.599 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.597+0000 7f8f997fa640 10 client.4589.objecter ms_handle_connect 0x7f8f8006dfd0 2024-03-25T18:32:11.609 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f8f94054960 con 0x7f8f9c1430c0 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 10 client.4589.objecter ms_dispatch 0x7f8f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f94050470 con 0x7f8f9c1430c0 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:11.610 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.608+0000 7f8f997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:11.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.841+0000 7f8fa2fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:11.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.841+0000 7f8fa2fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:11.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.841+0000 7f8fa2fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:11.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.841+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f9c14f330 con 0x7f8f9c1430c0 2024-03-25T18:32:11.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.897+0000 7f8f997fa640 1 -- 172.21.15.176:0/2903812553 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f8f9c14f330 con 0x7f8f9c1430c0 2024-03-25T18:32:11.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.897+0000 7f8f997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:11.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.897+0000 7f8f997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:11.899 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:11.899 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 10 librados: watch_flush enter 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 10 librados: watch_flush exit 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 10 client.4589.objecter successfully canceled tick 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 >> v1:172.21.15.176:6800/2598213983 conn(0x7f8f8006dfd0 legacy=0x7f8f80070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 10 monclient: shutdown 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:11.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 >> v1:172.21.15.176:6790/0 conn(0x7f8f9c1430c0 legacy=0x7f8f9c1e2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:11.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 shutdown_connections 2024-03-25T18:32:11.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 >> 172.21.15.176:0/2903812553 conn(0x7f8f9c0b7ea0 msgr2=0x7f8f9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:11.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 shutdown_connections 2024-03-25T18:32:11.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 -- 172.21.15.176:0/2903812553 wait complete. 2024-03-25T18:32:11.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:11.900+0000 7f8fa2fc8640 1 librados: shutdown 2024-03-25T18:32:11.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:18.720 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.832+0000 7f914a414640 10 monclient: get_monmap_and_config 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.832+0000 7f914a414640 10 monclient: build_initial_monmap 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.832+0000 7f914a414640 10 monclient: monmap: 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:18.832982+0000 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:18.832982+0000 2024-03-25T18:32:18.833 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:18.834 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:18.834 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:18.834 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:18.834 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:18.834 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.833+0000 7f914a414640 1 Processor -- start 2024-03-25T18:32:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.833+0000 7f914a414640 1 -- start start 2024-03-25T18:32:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.833+0000 7f914a414640 10 monclient: init 2024-03-25T18:32:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:18.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient(hunting): picked mon.noname-a con 0x7f91440a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient(hunting): picked mon.noname-b con 0x7f914413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient(hunting): picked mon.noname-c con 0x7f91441430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9144142520 con 0x7f91440a98f0 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440b28a0 con 0x7f91441430c0 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440b2a30 con 0x7f914413f710 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:18.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.834+0000 7f914a414640 10 monclient(hunting): authenticate will time out at 8617.146133s 2024-03-25T18:32:18.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.836+0000 7f91489b1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f91441430c0 0x7f914414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:51002/0 (socket says 172.21.15.176:51002) 2024-03-25T18:32:18.837 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.836+0000 7f91489b1640 1 -- 172.21.15.176:0/2032506845 learned_addr learned my addr 172.21.15.176:0/2032506845 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:18.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350623945 0 0) 0x7f91440b28a0 con 0x7f91441430c0 2024-03-25T18:32:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:32:18.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.850+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912c003520 con 0x7f91441430c0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241672672 0 0) 0x7f912c003520 con 0x7f91441430c0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 >> v1:172.21.15.176:6790/0 conn(0x7f914413f710 legacy=0x7f91441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 >> v1:172.21.15.176:6789/0 conn(0x7f91440a98f0 legacy=0x7f91440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 monclient: found mon.noname-c 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91440b44c0 con 0x7f91441430c0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:18.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f913c001a10 con 0x7f91441430c0 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f914a414640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f914a414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f9142ffd640 10 monclient: dump: 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:18.868 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:18.869 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:18.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.864+0000 7f914a414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f913c001be0 con 0x7f91441430c0 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 1 -- 172.21.15.176:0/2032506845 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f913c002030 con 0x7f91441430c0 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f914a414640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f914a414640 4 set_mon_vals no callback set 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f9142ffd640 10 monclient: dump: 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:18.870 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:18.871 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f914a414640 10 monclient: shutdown 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f914a414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.869+0000 7f914a414640 1 -- 172.21.15.176:0/2032506845 >> v1:172.21.15.186:6789/0 conn(0x7f91441430c0 legacy=0x7f914414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 1 -- 172.21.15.176:0/2032506845 shutdown_connections 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 1 -- 172.21.15.176:0/2032506845 >> 172.21.15.176:0/2032506845 conn(0x7f91440b7ea0 msgr2=0x7f91440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 1 -- 172.21.15.176:0/2032506845 shutdown_connections 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 1 -- 172.21.15.176:0/2032506845 wait complete. 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 10 monclient: build_initial_monmap 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.870+0000 7f914a414640 10 monclient: monmap: 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:18.873 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 librados: starting msgr at 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 librados: starting objecter 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 Processor -- start 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 -- start start 2024-03-25T18:32:18.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 librados: setting wanted keys 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 librados: calling monclient init 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient: init 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient(hunting): picked mon.noname-a con 0x7f91440a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient(hunting): picked mon.noname-c con 0x7f914413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient(hunting): picked mon.noname-b con 0x7f91441430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91441db7f0 con 0x7f91440a98f0 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9144138d30 con 0x7f91441430c0 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9144137e50 con 0x7f914413f710 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:18.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.871+0000 7f914a414640 10 monclient(hunting): authenticate will time out at 8617.183808s 2024-03-25T18:32:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.875+0000 7f91437fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f914413f710 0x7f91441db0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:54214/0 (socket says 172.21.15.176:54214) 2024-03-25T18:32:18.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.875+0000 7f91437fe640 1 -- 172.21.15.176:0/2649291129 learned_addr learned my addr 172.21.15.176:0/2649291129 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:18.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.877+0000 7f9140ff9640 10 client.?.objecter ms_handle_connect 0x7f914413f710 2024-03-25T18:32:18.878 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.877+0000 7f9140ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:18.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.886+0000 7f9140ff9640 10 client.?.objecter ms_handle_connect 0x7f91441430c0 2024-03-25T18:32:18.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.886+0000 7f9140ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:18.896 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:20.651 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:18.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.895+0000 7f9140ff9640 10 client.?.objecter ms_handle_connect 0x7f91440a98f0 2024-03-25T18:32:18.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.895+0000 7f9140ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:18.948 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:20.704 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864077208 0 0) 0x7f91441db7f0 con 0x7f91440a98f0 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:32:18.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.955+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130004eb0 con 0x7f91440a98f0 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793407086 0 0) 0x7f9130004eb0 con 0x7f91440a98f0 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91441db7f0 con 0x7f91440a98f0 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9134001870 con 0x7f91440a98f0 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:18.986 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:18.987 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:18.987 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:18.987 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:18.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:18.984+0000 7f9140ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:18.988 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:20.744 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:18.990 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:20.724 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.009 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:20.743 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2636238521 0 0) 0x7f91441db7f0 con 0x7f91440a98f0 2024-03-25T18:32:19.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 >> v1:172.21.15.176:6790/0 conn(0x7f914413f710 legacy=0x7f91441db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:19.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 >> v1:172.21.15.186:6789/0 conn(0x7f91441430c0 legacy=0x7f91441e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:19.011 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 1 monclient: found mon.a 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144137fe0 con 0x7f91440a98f0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 10 monclient: _finish_auth 0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f9140ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 10 monclient: _renew_subs 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9144137990 con 0x7f91440a98f0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 10 monclient: _renew_subs 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:19.012 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91441e3520 con 0x7f91440a98f0 2024-03-25T18:32:19.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.010+0000 7f914a414640 1 librados: init done 2024-03-25T18:32:19.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.011+0000 7f914a414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:19.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.011+0000 7f914a414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:19.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.011+0000 7f914a414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:19.013 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.011+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91440b3ef0 con 0x7f91440a98f0 2024-03-25T18:32:19.015 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:20.771 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.029 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:20.785 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.041 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:20.775 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.060 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:20.816 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91340035f0 con 0x7f91440a98f0 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9134003900 con 0x7f91440a98f0 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f9140ff9640 10 monclient: dump: 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:19.349 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.347+0000 7f91427fc640 4 set_mon_vals no callback set 2024-03-25T18:32:19.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.348+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f91340198d0 con 0x7f91440a98f0 2024-03-25T18:32:19.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.349+0000 7f91437fe640 10 monclient: get_auth_request con 0x7f913006dd20 auth_method 0 2024-03-25T18:32:19.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.349+0000 7f9140ff9640 10 client.4572.objecter ms_handle_connect 0x7f913006dd20 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f9134053600 con 0x7f91440a98f0 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 10 client.4572.objecter ms_dispatch 0x7f9144000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f913404ef40 con 0x7f91440a98f0 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:19.358 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.356+0000 7f9140ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:19.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.557+0000 7f914a414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:19.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.557+0000 7f914a414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:19.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.557+0000 7f914a414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:19.559 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.557+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91440627b0 con 0x7f91440a98f0 2024-03-25T18:32:19.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.596+0000 7f9140ff9640 1 -- 172.21.15.176:0/2649291129 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f9134051040 con 0x7f91440a98f0 2024-03-25T18:32:19.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.596+0000 7f9140ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:19.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.596+0000 7f9140ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:19.598 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:19.598 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:19.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 10 librados: watch_flush enter 2024-03-25T18:32:19.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 10 librados: watch_flush exit 2024-03-25T18:32:19.600 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 10 client.4572.objecter successfully canceled tick 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 >> v1:172.21.15.176:6800/2598213983 conn(0x7f913006dd20 legacy=0x7f91300701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 10 monclient: shutdown 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 >> v1:172.21.15.176:6789/0 conn(0x7f91440a98f0 legacy=0x7f9144138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 shutdown_connections 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 >> 172.21.15.176:0/2649291129 conn(0x7f91440b7ea0 msgr2=0x7f914413c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 shutdown_connections 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 -- 172.21.15.176:0/2649291129 wait complete. 2024-03-25T18:32:19.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:19.599+0000 7f914a414640 1 librados: shutdown 2024-03-25T18:32:19.613 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:26.420 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.532+0000 7ff3bb723640 10 monclient: get_monmap_and_config 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.532+0000 7ff3bb723640 10 monclient: build_initial_monmap 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.532+0000 7ff3bb723640 10 monclient: monmap: 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:26.533655+0000 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:26.533655+0000 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:26.534 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:26.535 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:26.535 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.533+0000 7ff3bb723640 1 Processor -- start 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.533+0000 7ff3bb723640 1 -- start start 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.533+0000 7ff3bb723640 10 monclient: init 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4142520 con 0x7ff3b41430c0 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40b28a0 con 0x7ff3b413f710 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40b2a30 con 0x7ff3b40a98f0 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:26.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.534+0000 7ff3bb723640 10 monclient(hunting): authenticate will time out at 8624.846451s 2024-03-25T18:32:26.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.538+0000 7ff3b94bf640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7ff3b40a98f0 0x7ff3b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:55194/0 (socket says 172.21.15.176:55194) 2024-03-25T18:32:26.540 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.539+0000 7ff3b94bf640 1 -- 172.21.15.176:0/744032411 learned_addr learned my addr 172.21.15.176:0/744032411 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:26.551 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469803523 0 0) 0x7ff3b40b28a0 con 0x7ff3b413f710 2024-03-25T18:32:26.551 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.551 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0003520 con 0x7ff3b413f710 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517423956 0 0) 0x7ff3b40b2a30 con 0x7ff3b40a98f0 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:32:26.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.550+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b40b28a0 con 0x7ff3b40a98f0 2024-03-25T18:32:26.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415618139 0 0) 0x7ff3b4142520 con 0x7ff3b41430c0 2024-03-25T18:32:26.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:32:26.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.554+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b40b2a30 con 0x7ff3b41430c0 2024-03-25T18:32:26.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613625955 0 0) 0x7ff3a0003520 con 0x7ff3b413f710 2024-03-25T18:32:26.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 >> v1:172.21.15.176:6790/0 conn(0x7ff3b40a98f0 legacy=0x7ff3b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:26.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 >> v1:172.21.15.176:6789/0 conn(0x7ff3b41430c0 legacy=0x7ff3b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:26.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:26.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 1 monclient: found mon.noname-c 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.558+0000 7ff3abfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b40b44c0 con 0x7ff3b413f710 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 10 monclient: _finish_auth 0 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff3a4001690 con 0x7ff3b413f710 2024-03-25T18:32:26.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3bb723640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3bb723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3abfff640 10 monclient: dump: 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:26.562 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:26.563 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:26.564 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.559+0000 7ff3bb723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:26.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3a40017f0 con 0x7ff3b413f710 2024-03-25T18:32:26.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 1 -- 172.21.15.176:0/744032411 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff3a4001c00 con 0x7ff3b413f710 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3bb723640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3bb723640 4 set_mon_vals no callback set 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.573+0000 7ff3abfff640 10 monclient: dump: 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:26.575 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:26.576 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:26.576 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:26.576 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 10 monclient: shutdown 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 1 -- 172.21.15.176:0/744032411 >> v1:172.21.15.186:6789/0 conn(0x7ff3b413f710 legacy=0x7ff3b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 1 -- 172.21.15.176:0/744032411 shutdown_connections 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 1 -- 172.21.15.176:0/744032411 >> 172.21.15.176:0/744032411 conn(0x7ff3b40b7ea0 msgr2=0x7ff3b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:26.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 1 -- 172.21.15.176:0/744032411 shutdown_connections 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.574+0000 7ff3bb723640 1 -- 172.21.15.176:0/744032411 wait complete. 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 10 monclient: build_initial_monmap 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 10 monclient: monmap: 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:26.579 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 librados: starting msgr at 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 librados: starting objecter 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 Processor -- start 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 -- start start 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 librados: setting wanted keys 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 1 librados: calling monclient init 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.575+0000 7ff3bb723640 10 monclient: init 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b40a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b41430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4137140 con 0x7ff3b413f710 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b41d7b90 con 0x7ff3b40a98f0 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b41d7590 con 0x7ff3b41430c0 2024-03-25T18:32:26.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:26.581 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.576+0000 7ff3bb723640 10 monclient(hunting): authenticate will time out at 8624.888399s 2024-03-25T18:32:26.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.589+0000 7ff3b94bf640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff3b40a98f0 0x7ff3b41394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:49880/0 (socket says 172.21.15.176:49880) 2024-03-25T18:32:26.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.589+0000 7ff3b94bf640 1 -- 172.21.15.176:0/3896250883 learned_addr learned my addr 172.21.15.176:0/3896250883 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:26.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.590+0000 7ff3a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3b40a98f0 2024-03-25T18:32:26.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.590+0000 7ff3a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.594+0000 7ff3a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3b41430c0 2024-03-25T18:32:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.594+0000 7ff3a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.594+0000 7ff3a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3b413f710 2024-03-25T18:32:26.595 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.594+0000 7ff3a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:26.601 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578639048 0 0) 0x7ff3b41d7590 con 0x7ff3b41430c0 2024-03-25T18:32:26.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:32:26.602 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.600+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff398004eb0 con 0x7ff3b41430c0 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193882074 0 0) 0x7ff3b41d7b90 con 0x7ff3b40a98f0 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:32:26.608 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.607+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b41d7590 con 0x7ff3b40a98f0 2024-03-25T18:32:26.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.615+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036020605 0 0) 0x7ff398004eb0 con 0x7ff3b41430c0 2024-03-25T18:32:26.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b41d7b90 con 0x7ff3b41430c0 2024-03-25T18:32:26.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff3ac001bd0 con 0x7ff3b41430c0 2024-03-25T18:32:26.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.618 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.619 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.616+0000 7ff3a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502938163 0 0) 0x7ff3b4137140 con 0x7ff3b413f710 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:32:26.620 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.617+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff398004eb0 con 0x7ff3b413f710 2024-03-25T18:32:26.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3653023897 0 0) 0x7ff3b41d7b90 con 0x7ff3b41430c0 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 >> v1:172.21.15.186:6789/0 conn(0x7ff3b40a98f0 legacy=0x7ff3b41394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 >> v1:172.21.15.176:6789/0 conn(0x7ff3b413f710 legacy=0x7ff3b4136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 1 monclient: found mon.c 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b41d7720 con 0x7ff3b41430c0 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.626+0000 7ff3a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:26.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 10 monclient: _renew_subs 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b413b7d0 con 0x7ff3b41430c0 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 10 monclient: _renew_subs 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b41d80b0 con 0x7ff3b41430c0 2024-03-25T18:32:26.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.627+0000 7ff3bb723640 1 librados: init done 2024-03-25T18:32:26.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.628+0000 7ff3bb723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:26.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.628+0000 7ff3bb723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:26.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.628+0000 7ff3bb723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:26.631 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.628+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b41d7d20 con 0x7ff3b41430c0 2024-03-25T18:32:26.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ac002060 con 0x7ff3b41430c0 2024-03-25T18:32:26.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:26.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7ff3ac003bf0 con 0x7ff3b41430c0 2024-03-25T18:32:26.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3a9ffb640 10 monclient: dump: 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:26.942 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:26.943 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.939+0000 7ff3ab7fe640 4 set_mon_vals no callback set 2024-03-25T18:32:26.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.940+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7ff3ac003e10 con 0x7ff3b41430c0 2024-03-25T18:32:26.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.941+0000 7ff3b94bf640 10 monclient: get_auth_request con 0x7ff39806de20 auth_method 0 2024-03-25T18:32:26.944 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.941+0000 7ff3a9ffb640 10 client.4601.objecter ms_handle_connect 0x7ff39806de20 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.945+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7ff3ac0538f0 con 0x7ff3b41430c0 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.945+0000 7ff3a9ffb640 10 client.4601.objecter ms_dispatch 0x7ff3b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.945+0000 7ff3a9ffb640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.946+0000 7ff3a9ffb640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.946+0000 7ff3a9ffb640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.946+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3ac04f400 con 0x7ff3b41430c0 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.946+0000 7ff3a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:26.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:26.946+0000 7ff3a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:27.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.189+0000 7ff3bb723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:27.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.189+0000 7ff3bb723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:27.190 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.189+0000 7ff3bb723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:27.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.189+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3b40627b0 con 0x7ff3b41430c0 2024-03-25T18:32:27.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.221+0000 7ff3a9ffb640 1 -- 172.21.15.176:0/3896250883 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7ff3ac051330 con 0x7ff3b41430c0 2024-03-25T18:32:27.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.221+0000 7ff3a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:27.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.221+0000 7ff3a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:27.224 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:27.224 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:27.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 10 librados: watch_flush enter 2024-03-25T18:32:27.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 10 librados: watch_flush exit 2024-03-25T18:32:27.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 >> v1:172.21.15.176:6800/2598213983 conn(0x7ff39806de20 legacy=0x7ff3980702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 10 monclient: shutdown 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 >> v1:172.21.15.176:6790/0 conn(0x7ff3b41430c0 legacy=0x7ff3b41e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 shutdown_connections 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 >> 172.21.15.176:0/3896250883 conn(0x7ff3b40b7ea0 msgr2=0x7ff3b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 shutdown_connections 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 -- 172.21.15.176:0/3896250883 wait complete. 2024-03-25T18:32:27.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:27.224+0000 7ff3bb723640 1 librados: shutdown 2024-03-25T18:32:27.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:34.044 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.162+0000 7f983cc63640 10 monclient: get_monmap_and_config 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.162+0000 7f983cc63640 10 monclient: build_initial_monmap 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.162+0000 7f983cc63640 10 monclient: monmap: 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:34.163 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:34.163209+0000 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:34.163209+0000 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:34.164 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.163+0000 7f983cc63640 1 Processor -- start 2024-03-25T18:32:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.163+0000 7f983cc63640 1 -- start start 2024-03-25T18:32:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.163+0000 7f983cc63640 10 monclient: init 2024-03-25T18:32:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.163+0000 7f983cc63640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:34.165 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.163+0000 7f983cc63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-c con 0x7f98380705b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-a con 0x7f98381402d0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-b con 0x7f9838149ae0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380718c0 con 0x7f98381402d0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380714c0 con 0x7f98380705b0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9838072b50 con 0x7f9838149ae0 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:34.166 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.164+0000 7f983cc63640 10 monclient(hunting): authenticate will time out at 8632.475999s 2024-03-25T18:32:34.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.174+0000 7f9835d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f98381402d0 0x7f9838140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54302/0 (socket says 172.21.15.176:54302) 2024-03-25T18:32:34.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.174+0000 7f9835d9b640 1 -- 172.21.15.176:0/1474792071 learned_addr learned my addr 172.21.15.176:0/1474792071 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359247592 0 0) 0x7f98380718c0 con 0x7f98381402d0 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:32:34.183 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.181+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003520 con 0x7f98381402d0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560484210 0 0) 0x7f9820003520 con 0x7f98381402d0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 legacy=0x7f983814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 >> v1:172.21.15.186:6789/0 conn(0x7f98380705b0 legacy=0x7f983813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 monclient: found mon.noname-a 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983813dd10 con 0x7f98381402d0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: _finish_auth 0 2024-03-25T18:32:34.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f982c001690 con 0x7f98381402d0 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983559a640 10 monclient: dump: 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:34.194 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:34.195 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983cc63640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:32:34.195 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.191+0000 7f983cc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:34.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f982c0017f0 con 0x7f98381402d0 2024-03-25T18:32:34.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:34.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 1 -- 172.21.15.176:0/1474792071 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f982c001c00 con 0x7f98381402d0 2024-03-25T18:32:34.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:34.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983559a640 10 monclient: dump: 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983cc63640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:34.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.207+0000 7f983cc63640 4 set_mon_vals no callback set 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 10 monclient: shutdown 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 1 -- 172.21.15.176:0/1474792071 >> v1:172.21.15.176:6789/0 conn(0x7f98381402d0 legacy=0x7f9838140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 1 -- 172.21.15.176:0/1474792071 shutdown_connections 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 1 -- 172.21.15.176:0/1474792071 >> 172.21.15.176:0/1474792071 conn(0x7f98380b5d10 msgr2=0x7f98380b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 1 -- 172.21.15.176:0/1474792071 shutdown_connections 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.208+0000 7f983cc63640 1 -- 172.21.15.176:0/1474792071 wait complete. 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 10 monclient: build_initial_monmap 2024-03-25T18:32:34.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 10 monclient: monmap: 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 librados: starting msgr at 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 librados: starting objecter 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 Processor -- start 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 -- start start 2024-03-25T18:32:34.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 librados: setting wanted keys 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 1 librados: calling monclient init 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.209+0000 7f983cc63640 10 monclient: init 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-a con 0x7f98380705b0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-b con 0x7f98381402d0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient(hunting): picked mon.noname-c con 0x7f9838149ae0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98381d95c0 con 0x7f98380705b0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98381d86b0 con 0x7f98381402d0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98381d8990 con 0x7f9838149ae0 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:34.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.210+0000 7f983cc63640 10 monclient(hunting): authenticate will time out at 8632.522119s 2024-03-25T18:32:34.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.215+0000 7f9836d9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 0x7f9838134d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:60212/0 (socket says 172.21.15.176:60212) 2024-03-25T18:32:34.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.215+0000 7f9836d9d640 1 -- 172.21.15.176:0/3676104070 learned_addr learned my addr 172.21.15.176:0/3676104070 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:34.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.217+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f9838149ae0 2024-03-25T18:32:34.218 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.217+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9836d9d640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 legacy=0x7f9838134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9836d9d640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 legacy=0x7f9838134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9836d9d640 1 --1- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 0x7f9838134d90 :-1 s=OPENED pgs=59910 cs=1 l=1).handle_message read tag failed 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9836d9d640 1 --1- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 0x7f9838134d90 :-1 s=OPENED pgs=59910 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9826ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6790/0 2024-03-25T18:32:34.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f98380705b0 2024-03-25T18:32:34.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.219+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:34.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676351418 0 0) 0x7f98381d95c0 con 0x7f98380705b0 2024-03-25T18:32:34.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:34.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:34.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:34.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:32:34.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.228+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98380705b0 2024-03-25T18:32:34.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.229+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f98381402d0 2024-03-25T18:32:34.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.229+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406398403 0 0) 0x7f9818004eb0 con 0x7f98380705b0 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98381d95c0 con 0x7f98380705b0 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f9828001930 con 0x7f98380705b0 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:34.240 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.238+0000 7f9826ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.239+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533660572 0 0) 0x7f98381d86b0 con 0x7f98381402d0 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.240+0000 7f9826ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:34.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.240+0000 7f9826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:34.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.240+0000 7f9826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:34.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.240+0000 7f9826ffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:32:34.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.240+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98381402d0 2024-03-25T18:32:34.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.248+0000 7f9835d9b640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.186:6789/0 conn(0x7f98381402d0 legacy=0x7f9838143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:32:34.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.248+0000 7f9835d9b640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.186:6789/0 conn(0x7f98381402d0 legacy=0x7f9838143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:32:34.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.248+0000 7f9835d9b640 1 --1- 172.21.15.176:0/3676104070 >> v1:172.21.15.186:6789/0 conn(0x7f98381402d0 0x7f9838143180 :-1 s=OPENED pgs=60520 cs=1 l=1).handle_message read tag failed 2024-03-25T18:32:34.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.248+0000 7f9835d9b640 1 --1- 172.21.15.176:0/3676104070 >> v1:172.21.15.186:6789/0 conn(0x7f98381402d0 0x7f9838143180 :-1 s=OPENED pgs=60520 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:32:34.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.248+0000 7f9826ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.186:6789/0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 312157181 0 0) 0x7f98381d95c0 con 0x7f98380705b0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6790/0 conn(0x7f9838149ae0 legacy=0x7f9838134d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.186:6789/0 conn(0x7f98381402d0 legacy=0x7f9838143180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 1 monclient: found mon.a 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838143890 con 0x7f98380705b0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f9826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:34.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 10 monclient: _renew_subs 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9838142ac0 con 0x7f98380705b0 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 10 monclient: _renew_subs 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f983813e960 con 0x7f98380705b0 2024-03-25T18:32:34.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.269+0000 7f983cc63640 1 librados: init done 2024-03-25T18:32:34.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.270+0000 7f983cc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:34.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.270+0000 7f983cc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:34.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.270+0000 7f983cc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.270+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98381d8b20 con 0x7f98380705b0 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9828003610 con 0x7f98380705b0 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f98280039a0 con 0x7f98380705b0 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9826ffd640 10 monclient: dump: 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.568 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.566+0000 7f9834d99640 4 set_mon_vals no callback set 2024-03-25T18:32:34.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.567+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f9828003bc0 con 0x7f98380705b0 2024-03-25T18:32:34.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.568+0000 7f9835d9b640 10 monclient: get_auth_request con 0x7f981806dd00 auth_method 0 2024-03-25T18:32:34.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.568+0000 7f9826ffd640 10 client.4584.objecter ms_handle_connect 0x7f981806dd00 2024-03-25T18:32:34.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f98280535a0 con 0x7f98380705b0 2024-03-25T18:32:34.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 10 client.4584.objecter ms_dispatch 0x7f9838000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:34.575 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:34.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:34.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T18:32:34.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f982804f0b0 con 0x7f98380705b0 2024-03-25T18:32:34.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:34.576 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.574+0000 7f9826ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:34.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.791+0000 7f983cc63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:34.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.791+0000 7f983cc63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:34.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.791+0000 7f983cc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:34.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.791+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9838143a20 con 0x7f98380705b0 2024-03-25T18:32:34.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.861+0000 7f9826ffd640 1 -- 172.21.15.176:0/3676104070 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f9828050fe0 con 0x7f98380705b0 2024-03-25T18:32:34.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.861+0000 7f9826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:34.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.861+0000 7f9826ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:34.865 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:34.865 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 10 librados: watch_flush enter 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 10 librados: watch_flush exit 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 10 client.4584.objecter successfully canceled tick 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6800/2598213983 conn(0x7f981806dd00 legacy=0x7f98180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 10 monclient: shutdown 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 >> v1:172.21.15.176:6789/0 conn(0x7f98380705b0 legacy=0x7f98381d8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 shutdown_connections 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 >> 172.21.15.176:0/3676104070 conn(0x7f98380b5d10 msgr2=0x7f983813c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 shutdown_connections 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 -- 172.21.15.176:0/3676104070 wait complete. 2024-03-25T18:32:34.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:34.863+0000 7f983cc63640 1 librados: shutdown 2024-03-25T18:32:34.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:39.039 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:40.795 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.044 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:40.799 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.088 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:08:40.844 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.093 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:40.827 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.097 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:40.853 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.106 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:40.840 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.106 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:40.862 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.124 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:40.858 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.133 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:08:40.889 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:41.682 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:41.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.797+0000 7f9206e8d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:41.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.797+0000 7f9206e8d640 10 monclient: build_initial_monmap 2024-03-25T18:32:41.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.797+0000 7f9206e8d640 10 monclient: monmap: 2024-03-25T18:32:41.798 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:41.798148+0000 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:41.798148+0000 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:41.799 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 1 Processor -- start 2024-03-25T18:32:41.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 1 -- start start 2024-03-25T18:32:41.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient: init 2024-03-25T18:32:41.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f92000b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f92000b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f920014be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92000af940 con 0x7f92000b4d60 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92000b0a70 con 0x7f92000b1730 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92000b0d50 con 0x7f920014be90 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:41.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.798+0000 7f9206e8d640 10 monclient(hunting): authenticate will time out at 8640.110596s 2024-03-25T18:32:41.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.805+0000 7f9204c29640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f92000b4d60 0x7f920013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:52550/0 (socket says 172.21.15.176:52550) 2024-03-25T18:32:41.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.805+0000 7f9204c29640 1 -- 172.21.15.176:0/799714011 learned_addr learned my addr 172.21.15.176:0/799714011 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300395163 0 0) 0x7f92000af940 con 0x7f92000b4d60 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:32:41.817 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.816+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0003520 con 0x7f92000b4d60 2024-03-25T18:32:41.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006140049 0 0) 0x7f92000b0d50 con 0x7f920014be90 2024-03-25T18:32:41.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:41.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:41.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:41.835 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:32:41.836 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.834+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92000af940 con 0x7f920014be90 2024-03-25T18:32:41.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703712918 0 0) 0x7f91f0003520 con 0x7f92000b4d60 2024-03-25T18:32:41.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 >> v1:172.21.15.176:6790/0 conn(0x7f920014be90 legacy=0x7f920014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:41.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 >> v1:172.21.15.186:6789/0 conn(0x7f92000b1730 legacy=0x7f92000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:41.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 monclient: found mon.noname-a 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92000b1030 con 0x7f92000b4d60 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f91f4001690 con 0x7f92000b4d60 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f9206e8d640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f9206e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f91ff7fe640 10 monclient: dump: 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:41.840 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.838+0000 7f9206e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91f40017f0 con 0x7f92000b4d60 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 1 -- 172.21.15.176:0/799714011 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f91f4001c00 con 0x7f92000b4d60 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f9206e8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f9206e8d640 4 set_mon_vals no callback set 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.860+0000 7f91ff7fe640 10 monclient: dump: 2024-03-25T18:32:41.861 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:41.862 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 10 monclient: shutdown 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 1 -- 172.21.15.176:0/799714011 >> v1:172.21.15.176:6789/0 conn(0x7f92000b4d60 legacy=0x7f920013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 1 -- 172.21.15.176:0/799714011 shutdown_connections 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 1 -- 172.21.15.176:0/799714011 >> 172.21.15.176:0/799714011 conn(0x7f92000617a0 msgr2=0x7f9200061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 1 -- 172.21.15.176:0/799714011 shutdown_connections 2024-03-25T18:32:41.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 1 -- 172.21.15.176:0/799714011 wait complete. 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 10 monclient: build_initial_monmap 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.861+0000 7f9206e8d640 10 monclient: monmap: 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 librados: starting msgr at 2024-03-25T18:32:41.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 librados: starting objecter 2024-03-25T18:32:41.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 Processor -- start 2024-03-25T18:32:41.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 -- start start 2024-03-25T18:32:41.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 librados: setting wanted keys 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 1 librados: calling monclient init 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 10 monclient: init 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f92000b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.862+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f92000b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f920014be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92001368f0 con 0x7f92000b1730 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9200137750 con 0x7f92000b4d60 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92001363d0 con 0x7f920014be90 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:41.868 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.863+0000 7f9206e8d640 10 monclient(hunting): authenticate will time out at 8640.174885s 2024-03-25T18:32:41.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.868+0000 7f920542a640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f920014be90 0x7f92001d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:58890/0 (socket says 172.21.15.176:58890) 2024-03-25T18:32:41.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.868+0000 7f920542a640 1 -- 172.21.15.176:0/932709491 learned_addr learned my addr 172.21.15.176:0/932709491 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:41.877 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.876+0000 7f91fd7fa640 10 client.?.objecter ms_handle_connect 0x7f920014be90 2024-03-25T18:32:41.877 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.876+0000 7f91fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:41.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.878+0000 7f91fd7fa640 10 client.?.objecter ms_handle_connect 0x7f92000b4d60 2024-03-25T18:32:41.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.878+0000 7f91fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:41.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.878+0000 7f91fd7fa640 10 client.?.objecter ms_handle_connect 0x7f92000b1730 2024-03-25T18:32:41.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.878+0000 7f91fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.884+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568752339 0 0) 0x7f92001368f0 con 0x7f92000b1730 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4004eb0 con 0x7f92000b1730 2024-03-25T18:32:41.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898779405 0 0) 0x7f9200137750 con 0x7f92000b4d60 2024-03-25T18:32:41.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:41.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:41.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:41.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:32:41.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.885+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92001368f0 con 0x7f92000b4d60 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547431054 0 0) 0x7f92001368f0 con 0x7f92000b4d60 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9200137750 con 0x7f92000b4d60 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f91ec0018b0 con 0x7f92000b4d60 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:41.895 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367670822 0 0) 0x7f91e4004eb0 con 0x7f92000b1730 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92001368f0 con 0x7f92000b1730 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f91f4001b00 con 0x7f92000b1730 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:41.896 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514998815 0 0) 0x7f92001363d0 con 0x7f920014be90 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:32:41.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.893+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4004eb0 con 0x7f920014be90 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3951846116 0 0) 0x7f9200137750 con 0x7f92000b4d60 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 >> v1:172.21.15.176:6790/0 conn(0x7f920014be90 legacy=0x7f92001d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 >> v1:172.21.15.176:6789/0 conn(0x7f92000b1730 legacy=0x7f9200135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 1 monclient: found mon.b 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200130470 con 0x7f92000b4d60 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f91fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:32:41.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 10 monclient: _renew_subs 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9200130720 con 0x7f92000b4d60 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 10 monclient: _renew_subs 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92001322f0 con 0x7f92000b4d60 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.898+0000 7f9206e8d640 1 librados: init done 2024-03-25T18:32:41.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.899+0000 7f9206e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:41.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.899+0000 7f9206e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:41.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.899+0000 7f9206e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:41.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:41.899+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9200136210 con 0x7f92000b4d60 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91ec003610 con 0x7f92000b4d60 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f91ec0039e0 con 0x7f92000b4d60 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91fd7fa640 10 monclient: dump: 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:42.185 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:42.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.183+0000 7f91feffd640 4 set_mon_vals no callback set 2024-03-25T18:32:42.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.184+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f91ec003c00 con 0x7f92000b4d60 2024-03-25T18:32:42.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.184+0000 7f9204c29640 10 monclient: get_auth_request con 0x7f91e406df00 auth_method 0 2024-03-25T18:32:42.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.185+0000 7f91fd7fa640 10 client.4603.objecter ms_handle_connect 0x7f91e406df00 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f91ec053520 con 0x7f92000b4d60 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 10 client.4603.objecter ms_dispatch 0x7f9200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.189+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91ec04f030 con 0x7f92000b4d60 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.190+0000 7f91fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:42.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.190+0000 7f91fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:42.411 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.410+0000 7f9206e8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:42.411 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.410+0000 7f9206e8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:42.411 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.410+0000 7f9206e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:32:42.411 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.410+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92000b2270 con 0x7f92000b4d60 2024-03-25T18:32:42.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.434+0000 7f91fd7fa640 1 -- 172.21.15.176:0/932709491 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f91ec050f60 con 0x7f92000b4d60 2024-03-25T18:32:42.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.434+0000 7f91fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:42.436 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.434+0000 7f91fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:42.436 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:42.436 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:42.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 10 librados: watch_flush enter 2024-03-25T18:32:42.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 10 librados: watch_flush exit 2024-03-25T18:32:42.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T18:32:42.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 10 client.4603.objecter successfully canceled tick 2024-03-25T18:32:42.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 >> v1:172.21.15.176:6800/2598213983 conn(0x7f91e406df00 legacy=0x7f91e40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 10 monclient: shutdown 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.436+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 >> v1:172.21.15.186:6789/0 conn(0x7f92000b4d60 legacy=0x7f920012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.437+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 shutdown_connections 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.437+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 >> 172.21.15.176:0/932709491 conn(0x7f92000617a0 msgr2=0x7f9200138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.437+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 shutdown_connections 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.437+0000 7f9206e8d640 1 -- 172.21.15.176:0/932709491 wait complete. 2024-03-25T18:32:42.439 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:42.437+0000 7f9206e8d640 1 librados: shutdown 2024-03-25T18:32:42.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:49.257 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:49.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.369+0000 7f3e40320640 10 monclient: get_monmap_and_config 2024-03-25T18:32:49.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.369+0000 7f3e40320640 10 monclient: build_initial_monmap 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.369+0000 7f3e40320640 10 monclient: monmap: 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:49.371250+0000 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:49.371250+0000 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:49.372 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.370+0000 7f3e40320640 1 Processor -- start 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.370+0000 7f3e40320640 1 -- start start 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.370+0000 7f3e40320640 10 monclient: init 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-b con 0x7f3e3813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-a con 0x7f3e381430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38142520 con 0x7f3e381430c0 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b28a0 con 0x7f3e380a98f0 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b2a30 con 0x7f3e3813f710 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:49.374 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.371+0000 7f3e40320640 10 monclient(hunting): authenticate will time out at 8647.684447s 2024-03-25T18:32:49.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.379+0000 7f3e3d8bb640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f3e3813f710 0x7f3e381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:49860/0 (socket says 172.21.15.176:49860) 2024-03-25T18:32:49.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.379+0000 7f3e3d8bb640 1 -- 172.21.15.176:0/1284260658 learned_addr learned my addr 172.21.15.176:0/1284260658 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723842786 0 0) 0x7f3e380b28a0 con 0x7f3e380a98f0 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 10 monclient(hunting): _negotiate 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:32:49.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.388+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24003520 con 0x7f3e380a98f0 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079007146 0 0) 0x7f3e380b2a30 con 0x7f3e3813f710 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 10 monclient(hunting): _negotiate 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:32:49.402 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.397+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e380b28a0 con 0x7f3e3813f710 2024-03-25T18:32:49.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769865835 0 0) 0x7f3e24003520 con 0x7f3e380a98f0 2024-03-25T18:32:49.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 >> v1:172.21.15.176:6790/0 conn(0x7f3e3813f710 legacy=0x7f3e381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:49.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 >> v1:172.21.15.176:6789/0 conn(0x7f3e381430c0 legacy=0x7f3e3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:49.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: _finish_hunting 0 2024-03-25T18:32:49.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 monclient: found mon.noname-c 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380b44c0 con 0x7f3e380a98f0 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: _finish_auth 0 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: _check_auth_tickets 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3e34001690 con 0x7f3e380a98f0 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e3d0ba640 10 monclient: dump: 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:49.407 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e40320640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:32:49.408 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.404+0000 7f3e40320640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.413+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e340017f0 con 0x7f3e380a98f0 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.413+0000 7f3e3d0ba640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.413+0000 7f3e40320640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.413+0000 7f3e40320640 4 set_mon_vals no callback set 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e3d0ba640 1 -- 172.21.15.176:0/1284260658 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3e34001c00 con 0x7f3e380a98f0 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e3d0ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e3d0ba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e3d0ba640 10 monclient: dump: 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.416 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:49.417 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 10 monclient: shutdown 2024-03-25T18:32:49.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:49.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 1 -- 172.21.15.176:0/1284260658 >> v1:172.21.15.186:6789/0 conn(0x7f3e380a98f0 legacy=0x7f3e380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:49.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 1 -- 172.21.15.176:0/1284260658 shutdown_connections 2024-03-25T18:32:49.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 1 -- 172.21.15.176:0/1284260658 >> 172.21.15.176:0/1284260658 conn(0x7f3e380b7ea0 msgr2=0x7f3e380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.414+0000 7f3e40320640 1 -- 172.21.15.176:0/1284260658 shutdown_connections 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.415+0000 7f3e40320640 1 -- 172.21.15.176:0/1284260658 wait complete. 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.415+0000 7f3e40320640 10 monclient: build_initial_monmap 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.415+0000 7f3e40320640 10 monclient: monmap: 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 librados: starting msgr at 2024-03-25T18:32:49.419 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 librados: starting objecter 2024-03-25T18:32:49.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 Processor -- start 2024-03-25T18:32:49.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 -- start start 2024-03-25T18:32:49.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 librados: setting wanted keys 2024-03-25T18:32:49.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 1 librados: calling monclient init 2024-03-25T18:32:49.420 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.416+0000 7f3e40320640 10 monclient: init 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-c con 0x7f3e3813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient(hunting): picked mon.noname-b con 0x7f3e381430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381d7500 con 0x7f3e380a98f0 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38139a10 con 0x7f3e381430c0 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381d8380 con 0x7f3e3813f710 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:49.421 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.417+0000 7f3e40320640 10 monclient(hunting): authenticate will time out at 8647.730255s 2024-03-25T18:32:49.433 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.431+0000 7f3e3e0bc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 0x7f3e38137ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:50854/0 (socket says 172.21.15.176:50854) 2024-03-25T18:32:49.433 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.431+0000 7f3e3e0bc640 1 -- 172.21.15.176:0/3633587154 learned_addr learned my addr 172.21.15.176:0/3633587154 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:49.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.432+0000 7f3e3e0bc640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 legacy=0x7f3e38137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:32:49.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.432+0000 7f3e3e0bc640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 legacy=0x7f3e38137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:32:49.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.432+0000 7f3e3e0bc640 1 --1- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 0x7f3e38137ac0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:32:49.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.432+0000 7f3e3e0bc640 1 --1- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 0x7f3e38137ac0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:32:49.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.432+0000 7f3e2effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6789/0 2024-03-25T18:32:49.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.436+0000 7f3e2effd640 10 client.?.objecter ms_handle_connect 0x7f3e3813f710 2024-03-25T18:32:49.438 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.436+0000 7f3e2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:49.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.449+0000 7f3e2effd640 10 client.?.objecter ms_handle_connect 0x7f3e381430c0 2024-03-25T18:32:49.451 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.449+0000 7f3e2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:49.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241439200 0 0) 0x7f3e381d8380 con 0x7f3e3813f710 2024-03-25T18:32:49.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:49.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:49.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:49.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:32:49.459 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.456+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c004eb0 con 0x7f3e3813f710 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586748283 0 0) 0x7f3e38139a10 con 0x7f3e381430c0 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:32:49.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.461+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e381d8380 con 0x7f3e381430c0 2024-03-25T18:32:49.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781857335 0 0) 0x7f3e1c004eb0 con 0x7f3e3813f710 2024-03-25T18:32:49.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e38139a10 con 0x7f3e3813f710 2024-03-25T18:32:49.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3e28001930 con 0x7f3e3813f710 2024-03-25T18:32:49.491 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 10 monclient(hunting): dump: 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.489+0000 7f3e2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578692772 0 0) 0x7f3e381d8380 con 0x7f3e381430c0 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e1c004eb0 con 0x7f3e381430c0 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3e30001b10 con 0x7f3e381430c0 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:49.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.506+0000 7f3e2effd640 10 monclient(hunting): dump: 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:49.509 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.518+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 249849499 0 0) 0x7f3e38139a10 con 0x7f3e3813f710 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.186:6789/0 conn(0x7f3e381430c0 legacy=0x7f3e38133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6789/0 conn(0x7f3e380a98f0 legacy=0x7f3e38137ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 1 monclient: found mon.c 2024-03-25T18:32:49.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e381d7750 con 0x7f3e3813f710 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 10 monclient: _finish_auth 0 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 10 monclient: _renew_subs 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e38134690 con 0x7f3e3813f710 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 10 monclient: _renew_subs 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381383f0 con 0x7f3e3813f710 2024-03-25T18:32:49.522 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.519+0000 7f3e40320640 1 librados: init done 2024-03-25T18:32:49.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.520+0000 7f3e40320640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:49.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.520+0000 7f3e40320640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:49.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.520+0000 7f3e40320640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:49.523 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.520+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e381d7990 con 0x7f3e3813f710 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e280047e0 con 0x7f3e3813f710 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f3e28004950 con 0x7f3e3813f710 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e2effd640 10 monclient: dump: 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:49.855 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.852+0000 7f3e3c8b9640 4 set_mon_vals no callback set 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.853+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f3e2801a920 con 0x7f3e3813f710 2024-03-25T18:32:49.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.853+0000 7f3e3e0bc640 10 monclient: get_auth_request con 0x7f3e1c06ddf0 auth_method 0 2024-03-25T18:32:49.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.854+0000 7f3e2effd640 10 client.4619.objecter ms_handle_connect 0x7f3e1c06ddf0 2024-03-25T18:32:49.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f3e28054670 con 0x7f3e3813f710 2024-03-25T18:32:49.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 10 client.4619.objecter ms_dispatch 0x7f3e38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:49.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:49.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:49.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T18:32:49.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e280b18b0 con 0x7f3e3813f710 2024-03-25T18:32:49.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:49.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:49.859+0000 7f3e2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:50.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.092+0000 7f3e40320640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:50.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.092+0000 7f3e40320640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:50.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.092+0000 7f3e40320640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:32:50.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.092+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e380627b0 con 0x7f3e3813f710 2024-03-25T18:32:50.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.140+0000 7f3e2effd640 1 -- 172.21.15.176:0/3633587154 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f3e28053d40 con 0x7f3e3813f710 2024-03-25T18:32:50.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.140+0000 7f3e2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:50.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.140+0000 7f3e2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:50.142 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:50.143 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:50.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.142+0000 7f3e40320640 10 librados: watch_flush enter 2024-03-25T18:32:50.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.142+0000 7f3e40320640 10 librados: watch_flush exit 2024-03-25T18:32:50.145 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.142+0000 7f3e40320640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.142+0000 7f3e40320640 10 client.4619.objecter successfully canceled tick 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6800/2598213983 conn(0x7f3e1c06ddf0 legacy=0x7f3e1c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 10 monclient: shutdown 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 >> v1:172.21.15.176:6790/0 conn(0x7f3e3813f710 legacy=0x7f3e381d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 shutdown_connections 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 >> 172.21.15.176:0/3633587154 conn(0x7f3e380b7ea0 msgr2=0x7f3e38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 shutdown_connections 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 -- 172.21.15.176:0/3633587154 wait complete. 2024-03-25T18:32:50.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:50.143+0000 7f3e40320640 1 librados: shutdown 2024-03-25T18:32:50.158 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:56.966 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.082+0000 7f77f554d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.082+0000 7f77f554d640 10 monclient: build_initial_monmap 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.082+0000 7f77f554d640 10 monclient: monmap: 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:32:57.084228+0000 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:32:57.084228+0000 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:57.085 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 1 Processor -- start 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 1 -- start start 2024-03-25T18:32:57.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient: init 2024-03-25T18:32:57.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:57.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-a con 0x7f77f00a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-b con 0x7f77f013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-c con 0x7f77f01430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f0142520 con 0x7f77f00a98f0 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f00b28a0 con 0x7f77f01430c0 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f00b2a30 con 0x7f77f013f710 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:57.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.083+0000 7f77f554d640 10 monclient(hunting): authenticate will time out at 8655.396558s 2024-03-25T18:32:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.090+0000 7f77eeffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f77f00a98f0 0x7f77f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:52080/0 (socket says 172.21.15.176:52080) 2024-03-25T18:32:57.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.090+0000 7f77eeffd640 1 -- 172.21.15.176:0/1063628043 learned_addr learned my addr 172.21.15.176:0/1063628043 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198200416 0 0) 0x7f77f0142520 con 0x7f77f00a98f0 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:32:57.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.106+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc003520 con 0x7f77f00a98f0 2024-03-25T18:32:57.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112363741 0 0) 0x7f77f00b2a30 con 0x7f77f013f710 2024-03-25T18:32:57.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:57.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:57.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:57.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:32:57.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.114+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f0142520 con 0x7f77f013f710 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 447022153 0 0) 0x7f77dc003520 con 0x7f77f00a98f0 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 >> v1:172.21.15.176:6790/0 conn(0x7f77f013f710 legacy=0x7f77f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 >> v1:172.21.15.186:6789/0 conn(0x7f77f01430c0 legacy=0x7f77f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 monclient: found mon.noname-a 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:57.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00b44c0 con 0x7f77f00a98f0 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77e0001690 con 0x7f77f00a98f0 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77f554d640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77f554d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77ee7fc640 10 monclient: dump: 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.119 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.115+0000 7f77f554d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:57.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77e00017f0 con 0x7f77f00a98f0 2024-03-25T18:32:57.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 1 -- 172.21.15.176:0/1063628043 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77e0001c00 con 0x7f77f00a98f0 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77f554d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77f554d640 4 set_mon_vals no callback set 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.125+0000 7f77ee7fc640 10 monclient: dump: 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:57.128 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:57.129 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:57.129 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 10 monclient: shutdown 2024-03-25T18:32:57.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:57.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 1 -- 172.21.15.176:0/1063628043 >> v1:172.21.15.176:6789/0 conn(0x7f77f00a98f0 legacy=0x7f77f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 1 -- 172.21.15.176:0/1063628043 shutdown_connections 2024-03-25T18:32:57.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 1 -- 172.21.15.176:0/1063628043 >> 172.21.15.176:0/1063628043 conn(0x7f77f00b7ea0 msgr2=0x7f77f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 1 -- 172.21.15.176:0/1063628043 shutdown_connections 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.126+0000 7f77f554d640 1 -- 172.21.15.176:0/1063628043 wait complete. 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.127+0000 7f77f554d640 10 monclient: build_initial_monmap 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.127+0000 7f77f554d640 10 monclient: monmap: 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:32:57.131 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 librados: starting msgr at 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 librados: starting objecter 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 Processor -- start 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 -- start start 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 librados: setting wanted keys 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 1 librados: calling monclient init 2024-03-25T18:32:57.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.128+0000 7f77f554d640 10 monclient: init 2024-03-25T18:32:57.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:57.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:57.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-a con 0x7f77f00a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:32:57.133 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-b con 0x7f77f013f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient(hunting): picked mon.noname-c con 0x7f77f01430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f01db050 con 0x7f77f00a98f0 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f01dab70 con 0x7f77f013f710 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f01dad00 con 0x7f77f01430c0 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:57.134 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.129+0000 7f77f554d640 10 monclient(hunting): authenticate will time out at 8655.442356s 2024-03-25T18:32:57.139 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.137+0000 7f77e67fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f77f013f710 0x7f77f01d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:36048/0 (socket says 172.21.15.176:36048) 2024-03-25T18:32:57.139 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.137+0000 7f77e67fc640 1 -- 172.21.15.176:0/700594542 learned_addr learned my addr 172.21.15.176:0/700594542 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:32:57.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.137+0000 7f77e7fff640 10 client.?.objecter ms_handle_connect 0x7f77f013f710 2024-03-25T18:32:57.140 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.138+0000 7f77e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.141+0000 7f77e7fff640 10 client.?.objecter ms_handle_connect 0x7f77f01430c0 2024-03-25T18:32:57.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.141+0000 7f77e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:57.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.155+0000 7f77e7fff640 10 client.?.objecter ms_handle_connect 0x7f77f00a98f0 2024-03-25T18:32:57.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.155+0000 7f77e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582070313 0 0) 0x7f77f01dab70 con 0x7f77f013f710 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:32:57.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.210+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d0004eb0 con 0x7f77f013f710 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340924131 0 0) 0x7f77f01db050 con 0x7f77f00a98f0 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:32:57.227 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.225+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f01dab70 con 0x7f77f00a98f0 2024-03-25T18:32:57.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654299656 0 0) 0x7f77d0004eb0 con 0x7f77f013f710 2024-03-25T18:32:57.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f01db050 con 0x7f77f013f710 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77d8001a10 con 0x7f77f013f710 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 10 monclient(hunting): dump: 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:57.243 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.240+0000 7f77e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269271400 0 0) 0x7f77f01dad00 con 0x7f77f01430c0 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:57.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:32:57.245 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.241+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d0004eb0 con 0x7f77f01430c0 2024-03-25T18:32:57.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102458862 0 0) 0x7f77f01dab70 con 0x7f77f00a98f0 2024-03-25T18:32:57.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f01dad00 con 0x7f77f00a98f0 2024-03-25T18:32:57.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77e0001b40 con 0x7f77f00a98f0 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.252+0000 7f77e7fff640 10 monclient(hunting): dump: 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:57.255 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120340655 0 0) 0x7f77f01dad00 con 0x7f77f00a98f0 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 >> v1:172.21.15.176:6790/0 conn(0x7f77f01430c0 legacy=0x7f77f01393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 >> v1:172.21.15.186:6789/0 conn(0x7f77f013f710 legacy=0x7f77f01d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 1 monclient: found mon.a 2024-03-25T18:32:57.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00b70a0 con 0x7f77f00a98f0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 10 monclient: _renew_subs 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f00b72f0 con 0x7f77f00a98f0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 10 monclient: _renew_subs 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f01e32a0 con 0x7f77f00a98f0 2024-03-25T18:32:57.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.256+0000 7f77f554d640 1 librados: init done 2024-03-25T18:32:57.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.257+0000 7f77f554d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:57.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.258+0000 7f77f554d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:57.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.258+0000 7f77f554d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.258+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f00627b0 con 0x7f77f00a98f0 2024-03-25T18:32:57.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.475+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77e0004a90 con 0x7f77f00a98f0 2024-03-25T18:32:57.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.475+0000 7f77e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:57.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.475+0000 7f77edffb640 4 set_mon_vals no callback set 2024-03-25T18:32:57.568 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.566+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77e0001250 con 0x7f77f00a98f0 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.566+0000 7f77e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.566+0000 7f77e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.566+0000 7f77e7fff640 10 monclient: dump: 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:32:57.569 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:32:57.570 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:32:57.570 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:32:57.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.567+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f77e001ac10 con 0x7f77f00a98f0 2024-03-25T18:32:57.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.568+0000 7f77e67fc640 10 monclient: get_auth_request con 0x7f77d006df70 auth_method 0 2024-03-25T18:32:57.570 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.568+0000 7f77e7fff640 10 client.4596.objecter ms_handle_connect 0x7f77d006df70 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.574+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f77e00548a0 con 0x7f77f00a98f0 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.574+0000 7f77e7fff640 10 client.4596.objecter ms_dispatch 0x7f77f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.574+0000 7f77e7fff640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.574+0000 7f77e7fff640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.575+0000 7f77e7fff640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.575+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77e00501e0 con 0x7f77f00a98f0 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.575+0000 7f77e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:57.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.575+0000 7f77e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:57.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.829+0000 7f77f554d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:57.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.829+0000 7f77f554d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:57.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.829+0000 7f77f554d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:32:57.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.829+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77f0074cc0 con 0x7f77f00a98f0 2024-03-25T18:32:57.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.852+0000 7f77e7fff640 1 -- 172.21.15.176:0/700594542 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f77e00522e0 con 0x7f77f00a98f0 2024-03-25T18:32:57.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.852+0000 7f77e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:57.854 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.852+0000 7f77e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:57.854 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:32:57.855 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 10 librados: watch_flush enter 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 10 librados: watch_flush exit 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 10 client.4596.objecter successfully canceled tick 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 >> v1:172.21.15.176:6800/2598213983 conn(0x7f77d006df70 legacy=0x7f77d0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 10 monclient: shutdown 2024-03-25T18:32:57.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 >> v1:172.21.15.176:6789/0 conn(0x7f77f00a98f0 legacy=0x7f77f01db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 shutdown_connections 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.854+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 >> 172.21.15.176:0/700594542 conn(0x7f77f00b7ea0 msgr2=0x7f77f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.855+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 shutdown_connections 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.855+0000 7f77f554d640 1 -- 172.21.15.176:0/700594542 wait complete. 2024-03-25T18:32:57.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:32:57.855+0000 7f77f554d640 1 librados: shutdown 2024-03-25T18:32:57.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:59.047 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:00.803 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.090 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:00.846 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.139 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:00.895 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.151 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:00.907 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.153 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:00.887 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.158 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:00.892 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.164 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:00.920 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.198 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:00.932 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:59.205 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:00.961 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.677 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.788+0000 7f7ee4dba640 10 monclient: get_monmap_and_config 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.788+0000 7f7ee4dba640 10 monclient: build_initial_monmap 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.788+0000 7f7ee4dba640 10 monclient: monmap: 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.790 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:04.790145+0000 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:04.790145+0000 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:04.791 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 1 Processor -- start 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 1 -- start start 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient: init 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:04.792 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee00b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee00b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee00af940 con 0x7f7ee014be90 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee00b0a70 con 0x7f7ee00b1730 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee00b0d50 con 0x7f7ee00b4d60 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:04.793 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.789+0000 7f7ee4dba640 10 monclient(hunting): authenticate will time out at 8663.102595s 2024-03-25T18:33:04.820 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.818+0000 7f7ed7fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7ee00b1730 0x7f7ee00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:54726/0 (socket says 172.21.15.176:54726) 2024-03-25T18:33:04.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.818+0000 7f7ed7fff640 1 -- 172.21.15.176:0/1231250614 learned_addr learned my addr 172.21.15.176:0/1231250614 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444082361 0 0) 0x7f7ee00b0a70 con 0x7f7ee00b1730 2024-03-25T18:33:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:04.831 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:04.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:04.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:33:04.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.829+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc003520 con 0x7f7ee00b1730 2024-03-25T18:33:04.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159329776 0 0) 0x7f7ee00b0d50 con 0x7f7ee00b4d60 2024-03-25T18:33:04.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:04.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:04.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:04.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:33:04.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.839+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee00b0a70 con 0x7f7ee00b4d60 2024-03-25T18:33:04.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925491095 0 0) 0x7f7ecc003520 con 0x7f7ee00b1730 2024-03-25T18:33:04.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 >> v1:172.21.15.176:6790/0 conn(0x7f7ee00b4d60 legacy=0x7f7ee013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:04.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 >> v1:172.21.15.176:6789/0 conn(0x7f7ee014be90 legacy=0x7f7ee014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:04.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:04.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 monclient: found mon.noname-c 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee00b1030 con 0x7f7ee00b1730 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ec8001690 con 0x7f7ee00b1730 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7eddd9b640 10 monclient: dump: 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.858 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7ee4dba640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:33:04.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.855+0000 7f7ee4dba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:04.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ec80017f0 con 0x7f7ee00b1730 2024-03-25T18:33:04.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:04.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 1 -- 172.21.15.176:0/1231250614 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ec8001c00 con 0x7f7ee00b1730 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7ee4dba640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7ee4dba640 4 set_mon_vals no callback set 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.864+0000 7f7eddd9b640 10 monclient: dump: 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:04.867 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 10 monclient: shutdown 2024-03-25T18:33:04.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:04.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 1 -- 172.21.15.176:0/1231250614 >> v1:172.21.15.186:6789/0 conn(0x7f7ee00b1730 legacy=0x7f7ee00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:04.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 1 -- 172.21.15.176:0/1231250614 shutdown_connections 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 1 -- 172.21.15.176:0/1231250614 >> 172.21.15.176:0/1231250614 conn(0x7f7ee00617a0 msgr2=0x7f7ee0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 1 -- 172.21.15.176:0/1231250614 shutdown_connections 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.865+0000 7f7ee4dba640 1 -- 172.21.15.176:0/1231250614 wait complete. 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 10 monclient: build_initial_monmap 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 10 monclient: monmap: 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 librados: starting msgr at 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 librados: starting objecter 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 Processor -- start 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 -- start start 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 librados: setting wanted keys 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 1 librados: calling monclient init 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.866+0000 7f7ee4dba640 10 monclient: init 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee00b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee00b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee014be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee01db590 con 0x7f7ee014be90 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee0141fc0 con 0x7f7ee00b4d60 2024-03-25T18:33:04.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee0140590 con 0x7f7ee00b1730 2024-03-25T18:33:04.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:04.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.867+0000 7f7ee4dba640 10 monclient(hunting): authenticate will time out at 8663.180011s 2024-03-25T18:33:04.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.871+0000 7f7eded9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7ee014be90 0x7f7ee01e3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:41664/0 (socket says 172.21.15.176:41664) 2024-03-25T18:33:04.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.871+0000 7f7eded9d640 1 -- 172.21.15.176:0/726384783 learned_addr learned my addr 172.21.15.176:0/726384783 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:04.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.872+0000 7f7ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ee014be90 2024-03-25T18:33:04.874 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.872+0000 7f7ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:04.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.873+0000 7f7ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ee00b1730 2024-03-25T18:33:04.875 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.873+0000 7f7ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:04.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.874+0000 7f7ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ee00b4d60 2024-03-25T18:33:04.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.874+0000 7f7ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:04.881 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161999142 0 0) 0x7f7ee01db590 con 0x7f7ee014be90 2024-03-25T18:33:04.881 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:04.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:04.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:04.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:33:04.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.879+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec0004eb0 con 0x7f7ee014be90 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263978010 0 0) 0x7f7ee0140590 con 0x7f7ee00b1730 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:33:04.887 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.885+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee01db590 con 0x7f7ee00b1730 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479892348 0 0) 0x7f7ee0141fc0 con 0x7f7ee00b4d60 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:33:04.889 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.887+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee0140590 con 0x7f7ee00b4d60 2024-03-25T18:33:04.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818818520 0 0) 0x7f7ee01db590 con 0x7f7ee00b1730 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ee0141fc0 con 0x7f7ee00b1730 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ed00018b0 con 0x7f7ee00b1730 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:04.899 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.896+0000 7f7ed6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280855631 0 0) 0x7f7ec0004eb0 con 0x7f7ee014be90 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ee01db590 con 0x7f7ee014be90 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ed8001c90 con 0x7f7ee014be90 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.900+0000 7f7ed6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:04.902 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:04.903 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:04.903 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1709517867 0 0) 0x7f7ee0140590 con 0x7f7ee00b4d60 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec0004eb0 con 0x7f7ee00b4d60 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ec8001b00 con 0x7f7ee00b4d60 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:04.906 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.904+0000 7f7ed6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:04.907 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4233402390 0 0) 0x7f7ee01db590 con 0x7f7ee014be90 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 >> v1:172.21.15.176:6790/0 conn(0x7f7ee00b1730 legacy=0x7f7ee01dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 >> v1:172.21.15.186:6789/0 conn(0x7f7ee00b4d60 legacy=0x7f7ee0134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 1 monclient: found mon.a 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee01d5440 con 0x7f7ee014be90 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:04.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 10 monclient: _renew_subs 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee0135550 con 0x7f7ee014be90 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 10 monclient: _renew_subs 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee0137c40 con 0x7f7ee014be90 2024-03-25T18:33:04.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.910+0000 7f7ee4dba640 1 librados: init done 2024-03-25T18:33:04.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.911+0000 7f7ee4dba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:04.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.911+0000 7f7ee4dba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:04.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.911+0000 7f7ee4dba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:04.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:04.911+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee0140720 con 0x7f7ee014be90 2024-03-25T18:33:05.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ed8003990 con 0x7f7ee014be90 2024-03-25T18:33:05.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:05.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7ed8003ce0 con 0x7f7ee014be90 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7ed6ffd640 10 monclient: dump: 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:05.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.207+0000 7f7edd59a640 4 set_mon_vals no callback set 2024-03-25T18:33:05.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.208+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f7ed8003f00 con 0x7f7ee014be90 2024-03-25T18:33:05.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.208+0000 7f7ede59c640 10 monclient: get_auth_request con 0x7f7ec006e180 auth_method 0 2024-03-25T18:33:05.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.208+0000 7f7ed6ffd640 10 client.4602.objecter ms_handle_connect 0x7f7ec006e180 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7ed8054a30 con 0x7f7ee014be90 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 10 client.4602.objecter ms_dispatch 0x7f7ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ed8050540 con 0x7f7ee014be90 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:05.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.214+0000 7f7ed6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:05.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.470+0000 7f7ee4dba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:05.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.470+0000 7f7ee4dba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:05.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.470+0000 7f7ee4dba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:05.473 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.470+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ee00627b0 con 0x7f7ee014be90 2024-03-25T18:33:05.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.493+0000 7f7ed6ffd640 1 -- 172.21.15.176:0/726384783 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7ed8052470 con 0x7f7ee014be90 2024-03-25T18:33:05.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.493+0000 7f7ed6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:05.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.493+0000 7f7ed6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:05.495 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:05.495 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 10 librados: watch_flush enter 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 10 librados: watch_flush exit 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 10 client.4602.objecter successfully canceled tick 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 >> v1:172.21.15.176:6800/2598213983 conn(0x7f7ec006e180 legacy=0x7f7ec0070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 10 monclient: shutdown 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 >> v1:172.21.15.176:6789/0 conn(0x7f7ee014be90 legacy=0x7f7ee01e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 shutdown_connections 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 >> 172.21.15.176:0/726384783 conn(0x7f7ee00617a0 msgr2=0x7f7ee013c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 shutdown_connections 2024-03-25T18:33:05.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 -- 172.21.15.176:0/726384783 wait complete. 2024-03-25T18:33:05.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:05.495+0000 7f7ee4dba640 1 librados: shutdown 2024-03-25T18:33:05.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:12.317 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.429+0000 7fe9d410d640 10 monclient: get_monmap_and_config 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.429+0000 7fe9d410d640 10 monclient: build_initial_monmap 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.430+0000 7fe9d410d640 10 monclient: monmap: 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:12.431843+0000 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:12.431843+0000 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:12.432 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:12.433 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:12.433 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.430+0000 7fe9d410d640 1 Processor -- start 2024-03-25T18:33:12.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.430+0000 7fe9d410d640 1 -- start start 2024-03-25T18:33:12.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.430+0000 7fe9d410d640 10 monclient: init 2024-03-25T18:33:12.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.431+0000 7fe9d410d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:12.434 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.431+0000 7fe9d410d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.431+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc0b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.431+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc0b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0af940 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0b0a70 con 0x7fe9cc0b1730 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0b0d50 con 0x7fe9cc14be90 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:12.435 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.432+0000 7fe9d410d640 10 monclient(hunting): authenticate will time out at 8670.745108s 2024-03-25T18:33:12.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.438+0000 7fe9d16a8640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe9cc0b1730 0x7fe9cc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:59620/0 (socket says 172.21.15.176:59620) 2024-03-25T18:33:12.440 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.438+0000 7fe9d16a8640 1 -- 172.21.15.176:0/3630091182 learned_addr learned my addr 172.21.15.176:0/3630091182 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:12.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237308750 0 0) 0x7fe9cc0af940 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.457 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b8003520 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432917207 0 0) 0x7fe9cc0b0a70 con 0x7fe9cc0b1730 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:33:12.458 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.455+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc0af940 con 0x7fe9cc0b1730 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.458+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501741742 0 0) 0x7fe9cc0b0d50 con 0x7fe9cc14be90 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.459+0000 7fe9d0ea7640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.459+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.459+0000 7fe9d0ea7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.459+0000 7fe9d0ea7640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:33:12.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.459+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc0b0a70 con 0x7fe9cc14be90 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663932731 0 0) 0x7fe9cc0af940 con 0x7fe9cc0b1730 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 >> v1:172.21.15.176:6790/0 conn(0x7fe9cc14be90 legacy=0x7fe9cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 >> v1:172.21.15.176:6789/0 conn(0x7fe9cc0b4d60 legacy=0x7fe9cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: _finish_hunting 0 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 monclient: found mon.noname-c 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0b1030 con 0x7fe9cc0b1730 2024-03-25T18:33:12.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: _finish_auth 0 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: _check_auth_tickets 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d410d640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe9bc001670 con 0x7fe9cc0b1730 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d0ea7640 10 monclient: dump: 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:12.475 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:12.476 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:12.476 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.472+0000 7fe9d410d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:12.505 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9bc0017d0 con 0x7fe9cc0b1730 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 1 -- 172.21.15.176:0/3630091182 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe9bc001be0 con 0x7fe9cc0b1730 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d410d640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d410d640 4 set_mon_vals no callback set 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.501+0000 7fe9d0ea7640 10 monclient: dump: 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.506 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 10 monclient: shutdown 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 1 -- 172.21.15.176:0/3630091182 >> v1:172.21.15.186:6789/0 conn(0x7fe9cc0b1730 legacy=0x7fe9cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 1 -- 172.21.15.176:0/3630091182 shutdown_connections 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 1 -- 172.21.15.176:0/3630091182 >> 172.21.15.176:0/3630091182 conn(0x7fe9cc0617a0 msgr2=0x7fe9cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 1 -- 172.21.15.176:0/3630091182 shutdown_connections 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.502+0000 7fe9d410d640 1 -- 172.21.15.176:0/3630091182 wait complete. 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 10 monclient: build_initial_monmap 2024-03-25T18:33:12.507 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 10 monclient: monmap: 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 1 librados: starting msgr at 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 1 librados: starting objecter 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 1 Processor -- start 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.503+0000 7fe9d410d640 1 -- start start 2024-03-25T18:33:12.508 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 1 librados: setting wanted keys 2024-03-25T18:33:12.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 1 librados: calling monclient init 2024-03-25T18:33:12.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient: init 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc0b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc14be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc140720 con 0x7fe9cc14be90 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc140380 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc13af80 con 0x7fe9cc0b1730 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:12.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.504+0000 7fe9d410d640 10 monclient(hunting): authenticate will time out at 8670.817517s 2024-03-25T18:33:12.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.515+0000 7fe9d1ea9640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fe9cc0b1730 0x7fe9cc134db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:44668/0 (socket says 172.21.15.176:44668) 2024-03-25T18:33:12.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.515+0000 7fe9d1ea9640 1 -- 172.21.15.176:0/3996038558 learned_addr learned my addr 172.21.15.176:0/3996038558 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:12.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.518+0000 7fe9c27fc640 10 client.?.objecter ms_handle_connect 0x7fe9cc0b1730 2024-03-25T18:33:12.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.518+0000 7fe9c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:12.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.524+0000 7fe9c27fc640 10 client.?.objecter ms_handle_connect 0x7fe9cc14be90 2024-03-25T18:33:12.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.524+0000 7fe9c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:12.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.524+0000 7fe9c27fc640 10 client.?.objecter ms_handle_connect 0x7fe9cc0b4d60 2024-03-25T18:33:12.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.524+0000 7fe9c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:12.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880476893 0 0) 0x7fe9cc140720 con 0x7fe9cc14be90 2024-03-25T18:33:12.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.534+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0004eb0 con 0x7fe9cc14be90 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887188943 0 0) 0x7fe9cc13af80 con 0x7fe9cc0b1730 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:33:12.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.535+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc140720 con 0x7fe9cc0b1730 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751066401 0 0) 0x7fe9cc140380 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:33:12.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.538+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc13af80 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1322177147 0 0) 0x7fe9cc13af80 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9cc140380 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe9bc001ae0 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 10 monclient(hunting): dump: 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:12.557 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.558 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.555+0000 7fe9c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:12.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1376299048 0 0) 0x7fe9cc140720 con 0x7fe9cc0b1730 2024-03-25T18:33:12.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9cc13af80 con 0x7fe9cc0b1730 2024-03-25T18:33:12.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe9c8001950 con 0x7fe9cc0b1730 2024-03-25T18:33:12.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:12.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient(hunting): dump: 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:12.563 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 106873577 0 0) 0x7fe9cc140380 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 >> v1:172.21.15.176:6790/0 conn(0x7fe9cc0b1730 legacy=0x7fe9cc134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 >> v1:172.21.15.176:6789/0 conn(0x7fe9cc14be90 legacy=0x7fe9cc1db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 monclient: found mon.b 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc13b110 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9d410d640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.560+0000 7fe9d410d640 10 monclient: _renew_subs 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc13bab0 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 monclient: _renew_subs 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1356a0 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 1 librados: init done 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.561+0000 7fe9d410d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.562+0000 7fe9d410d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:33:12.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.562+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc1401f0 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.581+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9bc004a90 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.581+0000 7fe9c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.581+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fe9bc001c60 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.582+0000 7fe9c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.582+0000 7fe9c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.582+0000 7fe9c27fc640 10 monclient: dump: 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:12.584 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.582+0000 7fe9c3fff640 4 set_mon_vals no callback set 2024-03-25T18:33:12.585 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.583+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fe9bc01ac10 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.583+0000 7fe9d1ea9640 10 monclient: get_auth_request con 0x7fe9b006df80 auth_method 0 2024-03-25T18:33:12.586 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.584+0000 7fe9c27fc640 10 client.4627.objecter ms_handle_connect 0x7fe9b006df80 2024-03-25T18:33:12.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.839+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fe9bc054800 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.839+0000 7fe9c27fc640 10 client.4627.objecter ms_dispatch 0x7fe9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:12.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.839+0000 7fe9c27fc640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:12.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.839+0000 7fe9c27fc640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:12.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.839+0000 7fe9c27fc640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:33:12.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.840+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9bc050310 con 0x7fe9cc0b4d60 2024-03-25T18:33:12.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.840+0000 7fe9c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:12.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:12.840+0000 7fe9c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:13.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.085+0000 7fe9d410d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:13.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.085+0000 7fe9d410d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:13.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.085+0000 7fe9d410d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:33:13.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.085+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9cc140510 con 0x7fe9cc0b4d60 2024-03-25T18:33:13.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.110+0000 7fe9c27fc640 1 -- 172.21.15.176:0/3996038558 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fe9bc052240 con 0x7fe9cc0b4d60 2024-03-25T18:33:13.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.110+0000 7fe9c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:13.112 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.110+0000 7fe9c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:13.112 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:13.113 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 10 librados: watch_flush enter 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 10 librados: watch_flush exit 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 >> v1:172.21.15.176:6800/2598213983 conn(0x7fe9b006df80 legacy=0x7fe9b0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 10 monclient: shutdown 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 >> v1:172.21.15.186:6789/0 conn(0x7fe9cc0b4d60 legacy=0x7fe9cc1dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 shutdown_connections 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 >> 172.21.15.176:0/3996038558 conn(0x7fe9cc0617a0 msgr2=0x7fe9cc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.112+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 shutdown_connections 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.113+0000 7fe9d410d640 1 -- 172.21.15.176:0/3996038558 wait complete. 2024-03-25T18:33:13.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:13.113+0000 7fe9d410d640 1 librados: shutdown 2024-03-25T18:33:13.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:19.111 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:20.866 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.163 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:20.919 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.200 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:20.934 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.206 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:20.962 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.211 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:20.945 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.236 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:20.992 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.243 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:20.977 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.244 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:21.000 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.279 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:21.035 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:19.933 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:20.047 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.046+0000 7fddbdd2b640 10 monclient: get_monmap_and_config 2024-03-25T18:33:20.047 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.046+0000 7fddbdd2b640 10 monclient: build_initial_monmap 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.046+0000 7fddbdd2b640 10 monclient: monmap: 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:20.047344+0000 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:20.047344+0000 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:20.048 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.047+0000 7fddbdd2b640 1 Processor -- start 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.047+0000 7fddbdd2b640 1 -- start start 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.047+0000 7fddbdd2b640 10 monclient: init 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:20.049 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-b con 0x7fddb80b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-a con 0x7fddb80b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-c con 0x7fddb814be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80af940 con 0x7fddb80b1730 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80b0a70 con 0x7fddb814be90 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80b0d50 con 0x7fddb80b4d60 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:20.050 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.048+0000 7fddbdd2b640 10 monclient(hunting): authenticate will time out at 8678.360494s 2024-03-25T18:33:20.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.058+0000 7fddaffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fddb80b1730 0x7fddb80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:38890/0 (socket says 172.21.15.176:38890) 2024-03-25T18:33:20.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.058+0000 7fddaffff640 1 -- 172.21.15.176:0/253728782 learned_addr learned my addr 172.21.15.176:0/253728782 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551717653 0 0) 0x7fddb80af940 con 0x7fddb80b1730 2024-03-25T18:33:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.071 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:33:20.072 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.070+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003520 con 0x7fddb80b1730 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126195058 0 0) 0x7fddb80b0a70 con 0x7fddb814be90 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:33:20.076 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.075+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb80af940 con 0x7fddb814be90 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560585209 0 0) 0x7fddb80b0d50 con 0x7fddb80b4d60 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:33:20.078 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.077+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb80b0a70 con 0x7fddb80b4d60 2024-03-25T18:33:20.079 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4224421741 0 0) 0x7fdda4003520 con 0x7fddb80b1730 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 >> v1:172.21.15.176:6790/0 conn(0x7fddb80b4d60 legacy=0x7fddb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 >> v1:172.21.15.186:6789/0 conn(0x7fddb814be90 legacy=0x7fddb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80b1030 con 0x7fddb80b1730 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdda0001690 con 0x7fddb80b1730 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddb6ffd640 10 monclient: dump: 2024-03-25T18:33:20.080 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddbdd2b640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:33:20.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.078+0000 7fddbdd2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdda0001870 con 0x7fddb80b1730 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 1 -- 172.21.15.176:0/253728782 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fdda0001c80 con 0x7fddb80b1730 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.087+0000 7fddb6ffd640 10 monclient: dump: 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:20.089 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:20.090 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:20.090 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:20.090 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:20.090 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 4 set_mon_vals no callback set 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 10 monclient: shutdown 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 1 -- 172.21.15.176:0/253728782 >> v1:172.21.15.176:6789/0 conn(0x7fddb80b1730 legacy=0x7fddb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.088+0000 7fddbdd2b640 1 -- 172.21.15.176:0/253728782 shutdown_connections 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.089+0000 7fddbdd2b640 1 -- 172.21.15.176:0/253728782 >> 172.21.15.176:0/253728782 conn(0x7fddb80617a0 msgr2=0x7fddb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:20.091 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.089+0000 7fddbdd2b640 1 -- 172.21.15.176:0/253728782 shutdown_connections 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.089+0000 7fddbdd2b640 1 -- 172.21.15.176:0/253728782 wait complete. 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.089+0000 7fddbdd2b640 10 monclient: build_initial_monmap 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.089+0000 7fddbdd2b640 10 monclient: monmap: 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 librados: starting msgr at 2024-03-25T18:33:20.092 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 librados: starting objecter 2024-03-25T18:33:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 Processor -- start 2024-03-25T18:33:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 -- start start 2024-03-25T18:33:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 librados: setting wanted keys 2024-03-25T18:33:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 1 librados: calling monclient init 2024-03-25T18:33:20.093 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.090+0000 7fddbdd2b640 10 monclient: init 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-b con 0x7fddb80b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-c con 0x7fddb80b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient(hunting): picked mon.noname-a con 0x7fddb814be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb8141e10 con 0x7fddb814be90 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb81d7960 con 0x7fddb80b1730 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb81d8590 con 0x7fddb80b4d60 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:20.094 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.091+0000 7fddbdd2b640 10 monclient(hunting): authenticate will time out at 8678.403110s 2024-03-25T18:33:20.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.104+0000 7fddb7fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fddb814be90 0x7fddb8142c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:38932/0 (socket says 172.21.15.176:38932) 2024-03-25T18:33:20.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.104+0000 7fddb7fff640 1 -- 172.21.15.176:0/2958491413 learned_addr learned my addr 172.21.15.176:0/2958491413 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:20.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.104+0000 7fddb4ff9640 10 client.?.objecter ms_handle_connect 0x7fddb814be90 2024-03-25T18:33:20.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.104+0000 7fddb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:20.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.107+0000 7fddb4ff9640 10 client.?.objecter ms_handle_connect 0x7fddb80b1730 2024-03-25T18:33:20.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.107+0000 7fddb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:20.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.112+0000 7fddb4ff9640 10 client.?.objecter ms_handle_connect 0x7fddb80b4d60 2024-03-25T18:33:20.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.112+0000 7fddb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749870351 0 0) 0x7fddb8141e10 con 0x7fddb814be90 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:33:20.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.116+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd98004eb0 con 0x7fddb814be90 2024-03-25T18:33:20.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476210242 0 0) 0x7fddb81d7960 con 0x7fddb80b1730 2024-03-25T18:33:20.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.124 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:33:20.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.123+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb8141e10 con 0x7fddb80b1730 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979829009 0 0) 0x7fddb81d8590 con 0x7fddb80b4d60 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:33:20.132 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.131+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb81d7960 con 0x7fddb80b4d60 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3663130999 0 0) 0x7fdd98004eb0 con 0x7fddb814be90 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb81d8590 con 0x7fddb814be90 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fddb0001c00 con 0x7fddb814be90 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:20.137 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.136+0000 7fddb4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:20.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 769483190 0 0) 0x7fddb81d8590 con 0x7fddb814be90 2024-03-25T18:33:20.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 >> v1:172.21.15.176:6790/0 conn(0x7fddb80b4d60 legacy=0x7fddb81d7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 >> v1:172.21.15.186:6789/0 conn(0x7fddb80b1730 legacy=0x7fddb8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:20.151 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 1 monclient: found mon.a 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb81d8720 con 0x7fddb814be90 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 10 monclient: _renew_subs 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb8142510 con 0x7fddb814be90 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 10 client.4614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 10 monclient: _renew_subs 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb8143360 con 0x7fddb814be90 2024-03-25T18:33:20.152 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.150+0000 7fddbdd2b640 1 librados: init done 2024-03-25T18:33:20.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.151+0000 7fddbdd2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:20.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.151+0000 7fddbdd2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:20.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.151+0000 7fddbdd2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.153 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.151+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb81d7af0 con 0x7fddb814be90 2024-03-25T18:33:20.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb00020d0 con 0x7fddb814be90 2024-03-25T18:33:20.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:20.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fddb0003b10 con 0x7fddb814be90 2024-03-25T18:33:20.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb4ff9640 10 monclient: dump: 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:20.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.461+0000 7fddb67fc640 4 set_mon_vals no callback set 2024-03-25T18:33:20.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.462+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fddb0003d30 con 0x7fddb814be90 2024-03-25T18:33:20.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.462+0000 7fddb77fe640 10 monclient: get_auth_request con 0x7fdd9806ddf0 auth_method 0 2024-03-25T18:33:20.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.462+0000 7fddb4ff9640 10 client.4614.objecter ms_handle_connect 0x7fdd9806ddf0 2024-03-25T18:33:20.467 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fddb00549d0 con 0x7fddb814be90 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 10 client.4614.objecter ms_dispatch 0x7fddb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 3 client.4614.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 3 client.4614.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 20 client.4614.objecter dump_active .. 0 homeless 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddb0050310 con 0x7fddb814be90 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:20.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.466+0000 7fddb4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:20.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.710+0000 7fddbdd2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:20.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.710+0000 7fddbdd2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:20.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.710+0000 7fddbdd2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:20.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.710+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddb80627b0 con 0x7fddb814be90 2024-03-25T18:33:20.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.732+0000 7fddb4ff9640 1 -- 172.21.15.176:0/2958491413 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fddb0052410 con 0x7fddb814be90 2024-03-25T18:33:20.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.732+0000 7fddb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:20.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.732+0000 7fddb4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:20.733 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:20.734 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 10 librados: watch_flush enter 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 10 librados: watch_flush exit 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 20 client.4614.objecter shutdown clearing up homeless session... 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 10 client.4614.objecter successfully canceled tick 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 >> v1:172.21.15.176:6800/2598213983 conn(0x7fdd9806ddf0 legacy=0x7fdd980702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 10 monclient: shutdown 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 >> v1:172.21.15.176:6789/0 conn(0x7fddb814be90 legacy=0x7fddb8142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 shutdown_connections 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 >> 172.21.15.176:0/2958491413 conn(0x7fddb80617a0 msgr2=0x7fddb813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 shutdown_connections 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 -- 172.21.15.176:0/2958491413 wait complete. 2024-03-25T18:33:20.737 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:20.735+0000 7fddbdd2b640 1 librados: shutdown 2024-03-25T18:33:20.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:27.557 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:27.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.665+0000 7f6f771ef640 10 monclient: get_monmap_and_config 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.665+0000 7f6f771ef640 10 monclient: build_initial_monmap 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.665+0000 7f6f771ef640 10 monclient: monmap: 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:27.666574+0000 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:27.666574+0000 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:27.667 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.666+0000 7f6f771ef640 1 Processor -- start 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.666+0000 7f6f771ef640 1 -- start start 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.666+0000 7f6f771ef640 10 monclient: init 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6f700b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-a con 0x7f6f700b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700af940 con 0x7f6f700b1730 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700b0a70 con 0x7f6f7014be90 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700b0d50 con 0x7f6f700b4d60 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:27.669 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.667+0000 7f6f771ef640 10 monclient(hunting): authenticate will time out at 8685.979266s 2024-03-25T18:33:27.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.677+0000 7f6f6ffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f6f700b1730 0x7f6f700b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:41740/0 (socket says 172.21.15.176:41740) 2024-03-25T18:33:27.678 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.677+0000 7f6f6ffff640 1 -- 172.21.15.176:0/2932940230 learned_addr learned my addr 172.21.15.176:0/2932940230 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161095044 0 0) 0x7f6f700af940 con 0x7f6f700b1730 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.687+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f60003520 con 0x7f6f700b1730 2024-03-25T18:33:27.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934869743 0 0) 0x7f6f700b0a70 con 0x7f6f7014be90 2024-03-25T18:33:27.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:27.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:27.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:27.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:33:27.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.688+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f700af940 con 0x7f6f7014be90 2024-03-25T18:33:27.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503793606 0 0) 0x7f6f60003520 con 0x7f6f700b1730 2024-03-25T18:33:27.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 >> v1:172.21.15.176:6790/0 conn(0x7f6f700b4d60 legacy=0x7f6f7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:27.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 >> v1:172.21.15.186:6789/0 conn(0x7f6f7014be90 legacy=0x7f6f7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:27.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:27.696 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700b1030 con 0x7f6f700b1730 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6f5c001690 con 0x7f6f700b1730 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f6f7fe640 10 monclient: dump: 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.697 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f771ef640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:33:27.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.695+0000 7f6f771ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f5c0017f0 con 0x7f6f700b1730 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 1 -- 172.21.15.176:0/2932940230 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6f5c001c00 con 0x7f6f700b1730 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f6f7fe640 10 monclient: dump: 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.707 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f771ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:27.708 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.706+0000 7f6f771ef640 4 set_mon_vals no callback set 2024-03-25T18:33:27.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 10 monclient: shutdown 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 1 -- 172.21.15.176:0/2932940230 >> v1:172.21.15.176:6789/0 conn(0x7f6f700b1730 legacy=0x7f6f700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 1 -- 172.21.15.176:0/2932940230 shutdown_connections 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 1 -- 172.21.15.176:0/2932940230 >> 172.21.15.176:0/2932940230 conn(0x7f6f700617a0 msgr2=0x7f6f70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 1 -- 172.21.15.176:0/2932940230 shutdown_connections 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.707+0000 7f6f771ef640 1 -- 172.21.15.176:0/2932940230 wait complete. 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 10 monclient: build_initial_monmap 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 10 monclient: monmap: 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:27.711 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 librados: starting msgr at 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 librados: starting objecter 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 Processor -- start 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 -- start start 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 librados: setting wanted keys 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 1 librados: calling monclient init 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.708+0000 7f6f771ef640 10 monclient: init 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-b con 0x7f6f700b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-a con 0x7f6f700b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7014be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:27.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70141f10 con 0x7f6f700b4d60 2024-03-25T18:33:27.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70140da0 con 0x7f6f700b1730 2024-03-25T18:33:27.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70138860 con 0x7f6f7014be90 2024-03-25T18:33:27.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:27.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.709+0000 7f6f771ef640 10 monclient(hunting): authenticate will time out at 8686.021216s 2024-03-25T18:33:27.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.719+0000 7f6f7578c640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f6f7014be90 0x7f6f701e2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:38568/0 (socket says 172.21.15.176:38568) 2024-03-25T18:33:27.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.719+0000 7f6f7578c640 1 -- 172.21.15.176:0/1958229434 learned_addr learned my addr 172.21.15.176:0/1958229434 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:27.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.724+0000 7f6f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f7014be90 2024-03-25T18:33:27.725 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.724+0000 7f6f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:27.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.727+0000 7f6f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f700b4d60 2024-03-25T18:33:27.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.727+0000 7f6f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.733+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512343195 0 0) 0x7f6f70141f10 con 0x7f6f700b4d60 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f54004eb0 con 0x7f6f700b4d60 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f700b1730 2024-03-25T18:33:27.735 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.734+0000 7f6f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:27.744 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839718178 0 0) 0x7f6f54004eb0 con 0x7f6f700b4d60 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f70141f10 con 0x7f6f700b4d60 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6f5c001bc0 con 0x7f6f700b4d60 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:27.745 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.743+0000 7f6f6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907883290 0 0) 0x7f6f70140da0 con 0x7f6f700b1730 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:33:27.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.744+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f54004eb0 con 0x7f6f700b1730 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603161417 0 0) 0x7f6f70138860 con 0x7f6f7014be90 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:33:27.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.746+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70140da0 con 0x7f6f7014be90 2024-03-25T18:33:27.762 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 859623741 0 0) 0x7f6f54004eb0 con 0x7f6f700b1730 2024-03-25T18:33:27.762 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f70138860 con 0x7f6f700b1730 2024-03-25T18:33:27.762 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6f64001860 con 0x7f6f700b1730 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.761+0000 7f6f6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:27.763 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3626523467 0 0) 0x7f6f70141f10 con 0x7f6f700b4d60 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 >> v1:172.21.15.176:6790/0 conn(0x7f6f7014be90 legacy=0x7f6f701e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 >> v1:172.21.15.186:6789/0 conn(0x7f6f700b1730 legacy=0x7f6f701405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 1 monclient: found mon.a 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70137f50 con 0x7f6f700b4d60 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:27.774 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 5 monclient: authenticate success, global_id 4620 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 10 monclient: _renew_subs 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f701d8080 con 0x7f6f700b4d60 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 10 monclient: _renew_subs 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f701d8560 con 0x7f6f700b4d60 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.772+0000 7f6f771ef640 1 librados: init done 2024-03-25T18:33:27.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.774+0000 7f6f771ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:27.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.774+0000 7f6f771ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:27.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.774+0000 7f6f771ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:27.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:27.774+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f701381a0 con 0x7f6f700b4d60 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f5c004a90 con 0x7f6f700b4d60 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6f5c004da0 con 0x7f6f700b4d60 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6d7fa640 10 monclient: dump: 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:28.068 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:28.069 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.066+0000 7f6f6effd640 4 set_mon_vals no callback set 2024-03-25T18:33:28.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.067+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f6f5c01ad70 con 0x7f6f700b4d60 2024-03-25T18:33:28.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.068+0000 7f6f74f8b640 10 monclient: get_auth_request con 0x7f6f5406e020 auth_method 0 2024-03-25T18:33:28.070 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.068+0000 7f6f6d7fa640 10 client.4620.objecter ms_handle_connect 0x7f6f5406e020 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f6f5c054980 con 0x7f6f700b4d60 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 10 client.4620.objecter ms_dispatch 0x7f6f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f5c050490 con 0x7f6f700b4d60 2024-03-25T18:33:28.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:28.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.073+0000 7f6f6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:28.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.326+0000 7f6f771ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:28.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.326+0000 7f6f771ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:28.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.326+0000 7f6f771ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:28.328 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.326+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f700627b0 con 0x7f6f700b4d60 2024-03-25T18:33:28.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.348+0000 7f6f6d7fa640 1 -- 172.21.15.176:0/1958229434 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f6f5c0523c0 con 0x7f6f700b4d60 2024-03-25T18:33:28.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.348+0000 7f6f6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:28.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.348+0000 7f6f6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:28.349 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:28.350 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 10 librados: watch_flush enter 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 10 librados: watch_flush exit 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 10 client.4620.objecter successfully canceled tick 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 >> v1:172.21.15.176:6800/2598213983 conn(0x7f6f5406e020 legacy=0x7f6f540704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 10 monclient: shutdown 2024-03-25T18:33:28.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 >> v1:172.21.15.176:6789/0 conn(0x7f6f700b4d60 legacy=0x7f6f701d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 shutdown_connections 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 >> 172.21.15.176:0/1958229434 conn(0x7f6f700617a0 msgr2=0x7f6f700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 shutdown_connections 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 -- 172.21.15.176:0/1958229434 wait complete. 2024-03-25T18:33:28.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:28.350+0000 7f6f771ef640 1 librados: shutdown 2024-03-25T18:33:28.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:35.171 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:35.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.282+0000 7f7816e4e640 10 monclient: get_monmap_and_config 2024-03-25T18:33:35.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.282+0000 7f7816e4e640 10 monclient: build_initial_monmap 2024-03-25T18:33:35.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.282+0000 7f7816e4e640 10 monclient: monmap: 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:35.283494+0000 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:35.283494+0000 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:35.284 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.283+0000 7f7816e4e640 1 Processor -- start 2024-03-25T18:33:35.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.283+0000 7f7816e4e640 1 -- start start 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.283+0000 7f7816e4e640 10 monclient: init 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f78100b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f78100b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f781014be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100af940 con 0x7f78100b4d60 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100b0a70 con 0x7f78100b1730 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100b0d50 con 0x7f781014be90 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:35.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.284+0000 7f7816e4e640 10 monclient(hunting): authenticate will time out at 8693.596463s 2024-03-25T18:33:35.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.292+0000 7f780ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f78100b1730 0x7f78100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:53102/0 (socket says 172.21.15.176:53102) 2024-03-25T18:33:35.293 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.292+0000 7f780ffff640 1 -- 172.21.15.176:0/338557374 learned_addr learned my addr 172.21.15.176:0/338557374 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48088167 0 0) 0x7f78100b0a70 con 0x7f78100b1730 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.307+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7800003520 con 0x7f78100b1730 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264545763 0 0) 0x7f78100af940 con 0x7f78100b4d60 2024-03-25T18:33:35.309 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:35.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:35.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:35.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:33:35.310 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.308+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78100b0a70 con 0x7f78100b4d60 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568500801 0 0) 0x7f78100b0a70 con 0x7f78100b4d60 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 >> v1:172.21.15.176:6790/0 conn(0x7f781014be90 legacy=0x7f781014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 >> v1:172.21.15.186:6789/0 conn(0x7f78100b1730 legacy=0x7f78100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:35.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78100b1030 con 0x7f78100b4d60 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7804001670 con 0x7f78100b4d60 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.317+0000 7f780f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.318+0000 7f780f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.318+0000 7f780f7fe640 10 monclient: dump: 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:35.320 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.318+0000 7f7816e4e640 5 monclient: authenticate success, global_id 4623 2024-03-25T18:33:35.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.318+0000 7f7816e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78040017d0 con 0x7f78100b4d60 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 1 -- 172.21.15.176:0/338557374 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7804001ba0 con 0x7f78100b4d60 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f7816e4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f7816e4e640 4 set_mon_vals no callback set 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.330+0000 7f780f7fe640 10 monclient: dump: 2024-03-25T18:33:35.331 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:35.332 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 10 monclient: shutdown 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 1 -- 172.21.15.176:0/338557374 >> v1:172.21.15.176:6789/0 conn(0x7f78100b4d60 legacy=0x7f781013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 1 -- 172.21.15.176:0/338557374 shutdown_connections 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 1 -- 172.21.15.176:0/338557374 >> 172.21.15.176:0/338557374 conn(0x7f78100617a0 msgr2=0x7f7810061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:35.333 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 1 -- 172.21.15.176:0/338557374 shutdown_connections 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.331+0000 7f7816e4e640 1 -- 172.21.15.176:0/338557374 wait complete. 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.332+0000 7f7816e4e640 10 monclient: build_initial_monmap 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.332+0000 7f7816e4e640 10 monclient: monmap: 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:35.334 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:35.335 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.332+0000 7f7816e4e640 1 librados: starting msgr at 2024-03-25T18:33:35.335 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.332+0000 7f7816e4e640 1 librados: starting objecter 2024-03-25T18:33:35.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 Processor -- start 2024-03-25T18:33:35.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 -- start start 2024-03-25T18:33:35.336 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 librados: setting wanted keys 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 librados: calling monclient init 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient: init 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f78100b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f78100b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f781014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7810137e90 con 0x7f78100b4d60 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.333+0000 7f7816e4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7810138e60 con 0x7f781014be90 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.334+0000 7f7816e4e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7810139180 con 0x7f78100b1730 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.334+0000 7f7816e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:35.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.334+0000 7f7816e4e640 10 monclient(hunting): authenticate will time out at 8693.645859s 2024-03-25T18:33:35.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.345+0000 7f7814bea640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f78100b1730 0x7f7810131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:59508/0 (socket says 172.21.15.176:59508) 2024-03-25T18:33:35.346 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.345+0000 7f7814bea640 1 -- 172.21.15.176:0/705866147 learned_addr learned my addr 172.21.15.176:0/705866147 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:35.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.347+0000 7f780d7fa640 10 client.?.objecter ms_handle_connect 0x7f78100b1730 2024-03-25T18:33:35.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.347+0000 7f780d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:35.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.347+0000 7f780d7fa640 10 client.?.objecter ms_handle_connect 0x7f78100b4d60 2024-03-25T18:33:35.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.347+0000 7f780d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:35.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.354+0000 7f780d7fa640 10 client.?.objecter ms_handle_connect 0x7f781014be90 2024-03-25T18:33:35.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.354+0000 7f780d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173073474 0 0) 0x7f7810139180 con 0x7f78100b1730 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:33:35.364 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.363+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f4004eb0 con 0x7f78100b1730 2024-03-25T18:33:35.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016131268 0 0) 0x7f7810137e90 con 0x7f78100b4d60 2024-03-25T18:33:35.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:35.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:35.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:35.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:33:35.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.369+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810139180 con 0x7f78100b4d60 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.371+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142451140 0 0) 0x7f7810138e60 con 0x7f781014be90 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.371+0000 7f780d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.371+0000 7f780d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.371+0000 7f780d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.372+0000 7f780d7fa640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:33:35.373 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.372+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810137e90 con 0x7f781014be90 2024-03-25T18:33:35.378 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.377+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 711983918 0 0) 0x7f77f4004eb0 con 0x7f78100b1730 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.377+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7810138e60 con 0x7f78100b1730 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.378+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f78040035f0 con 0x7f78100b1730 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.378+0000 7f780d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.378+0000 7f780d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.378+0000 7f780d7fa640 10 monclient(hunting): dump: 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:35.379 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:35.380 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.380 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.378+0000 7f780d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:35.382 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615874688 0 0) 0x7f7810139180 con 0x7f78100b4d60 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f4004eb0 con 0x7f78100b4d60 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f77fc0018d0 con 0x7f78100b4d60 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.381+0000 7f780d7fa640 10 monclient(hunting): dump: 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:35.383 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:35.384 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.394 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2663032094 0 0) 0x7f7810138e60 con 0x7f78100b1730 2024-03-25T18:33:35.394 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 >> v1:172.21.15.186:6789/0 conn(0x7f781014be90 legacy=0x7f78101e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.394 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 >> v1:172.21.15.176:6789/0 conn(0x7f78100b4d60 legacy=0x7f781013af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.394 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 1 monclient: found mon.c 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7810139310 con 0x7f78100b1730 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f780d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 10 monclient: _renew_subs 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7810138470 con 0x7f78100b1730 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 10 monclient: _renew_subs 2024-03-25T18:33:35.395 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7810132700 con 0x7f78100b1730 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.392+0000 7f7816e4e640 1 librados: init done 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.394+0000 7f7816e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.394+0000 7f7816e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.394+0000 7f7816e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:35.396 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.394+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78101401f0 con 0x7f78100b1730 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7804003ac0 con 0x7f78100b1730 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f7804003ed0 con 0x7f78100b1730 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780d7fa640 10 monclient: dump: 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:35.693 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:35.694 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.691+0000 7f780effd640 4 set_mon_vals no callback set 2024-03-25T18:33:35.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.692+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f78040040f0 con 0x7f78100b1730 2024-03-25T18:33:35.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.693+0000 7f780ffff640 10 monclient: get_auth_request con 0x7f77f406df70 auth_method 0 2024-03-25T18:33:35.695 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.693+0000 7f780d7fa640 10 client.4655.objecter ms_handle_connect 0x7f77f406df70 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f7804001940 con 0x7f78100b1730 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 10 client.4655.objecter ms_dispatch 0x7f7810000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78040504e0 con 0x7f78100b1730 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:35.701 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.699+0000 7f780d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:35.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.940+0000 7f7816e4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:35.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.940+0000 7f7816e4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:35.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.940+0000 7f7816e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:35.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.940+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78100627b0 con 0x7f78100b1730 2024-03-25T18:33:35.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.980+0000 7f780d7fa640 1 -- 172.21.15.176:0/705866147 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f7804052410 con 0x7f78100b1730 2024-03-25T18:33:35.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.980+0000 7f780d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:35.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.980+0000 7f780d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:35.981 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:35.981 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 10 librados: watch_flush enter 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 10 librados: watch_flush exit 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 10 client.4655.objecter successfully canceled tick 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 >> v1:172.21.15.176:6800/2598213983 conn(0x7f77f406df70 legacy=0x7f77f4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 10 monclient: shutdown 2024-03-25T18:33:35.984 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 >> v1:172.21.15.176:6790/0 conn(0x7f78100b1730 legacy=0x7f7810131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 shutdown_connections 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.982+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 >> 172.21.15.176:0/705866147 conn(0x7f78100617a0 msgr2=0x7f781013c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.983+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 shutdown_connections 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.983+0000 7f7816e4e640 1 -- 172.21.15.176:0/705866147 wait complete. 2024-03-25T18:33:35.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:35.983+0000 7f7816e4e640 1 librados: shutdown 2024-03-25T18:33:35.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:39.181 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:40.937 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.234 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:40.990 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.270 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:09:41.026 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.284 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:41.040 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.289 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:41.024 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.291 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:41.047 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.292 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:41.026 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.314 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:41.049 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:39.318 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:09:41.074 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.803 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.918+0000 7f41b9797640 10 monclient: get_monmap_and_config 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.918+0000 7f41b9797640 10 monclient: build_initial_monmap 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.918+0000 7f41b9797640 10 monclient: monmap: 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:42.919233+0000 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:42.919233+0000 2024-03-25T18:33:42.919 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:42.920 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:42.920 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:42.920 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:42.920 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:42.920 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.919+0000 7f41b9797640 1 Processor -- start 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.919+0000 7f41b9797640 1 -- start start 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.919+0000 7f41b9797640 10 monclient: init 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.919+0000 7f41b9797640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.919+0000 7f41b9797640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-b con 0x7f41b40a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-a con 0x7f41b413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:42.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-c con 0x7f41b41430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:42.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b4142520 con 0x7f41b413f710 2024-03-25T18:33:42.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40b28a0 con 0x7f41b41430c0 2024-03-25T18:33:42.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40b2a30 con 0x7f41b40a98f0 2024-03-25T18:33:42.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:42.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.920+0000 7f41b9797640 10 monclient(hunting): authenticate will time out at 8701.232092s 2024-03-25T18:33:42.934 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.933+0000 7f41b2ffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f41b40a98f0 0x7f41b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:36008/0 (socket says 172.21.15.176:36008) 2024-03-25T18:33:42.934 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.933+0000 7f41b2ffd640 1 -- 172.21.15.176:0/1223041463 learned_addr learned my addr 172.21.15.176:0/1223041463 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:42.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203284118 0 0) 0x7f41b4142520 con 0x7f41b413f710 2024-03-25T18:33:42.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.954+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0003520 con 0x7f41b413f710 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411843632 0 0) 0x7f41b40b28a0 con 0x7f41b41430c0 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:42.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.955+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b4142520 con 0x7f41b41430c0 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153550757 0 0) 0x7f41b40b2a30 con 0x7f41b40a98f0 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:42.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:42.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:42.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:33:42.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.956+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b40b28a0 con 0x7f41b40a98f0 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.983+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2935092947 0 0) 0x7f41a0003520 con 0x7f41b413f710 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.983+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 >> v1:172.21.15.176:6790/0 conn(0x7f41b40a98f0 legacy=0x7f41b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.983+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 >> v1:172.21.15.186:6789/0 conn(0x7f41b41430c0 legacy=0x7f41b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.983+0000 7f41b27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.983+0000 7f41b27fc640 1 monclient: found mon.noname-a 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:33:42.985 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40b44c0 con 0x7f41b413f710 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f419c001690 con 0x7f41b413f710 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b9797640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b9797640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b27fc640 10 monclient: dump: 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:42.986 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:42.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.984+0000 7f41b9797640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f419c0017f0 con 0x7f41b413f710 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 1 -- 172.21.15.176:0/1223041463 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f419c001c00 con 0x7f41b413f710 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b9797640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b9797640 4 set_mon_vals no callback set 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b27fc640 10 monclient: dump: 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:42.997 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:42.998 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b9797640 10 monclient: shutdown 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b9797640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.996+0000 7f41b9797640 1 -- 172.21.15.176:0/1223041463 >> v1:172.21.15.176:6789/0 conn(0x7f41b413f710 legacy=0x7f41b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.997+0000 7f41b9797640 1 -- 172.21.15.176:0/1223041463 shutdown_connections 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.997+0000 7f41b9797640 1 -- 172.21.15.176:0/1223041463 >> 172.21.15.176:0/1223041463 conn(0x7f41b40b7ea0 msgr2=0x7f41b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.997+0000 7f41b9797640 1 -- 172.21.15.176:0/1223041463 shutdown_connections 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.997+0000 7f41b9797640 1 -- 172.21.15.176:0/1223041463 wait complete. 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 10 monclient: build_initial_monmap 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 10 monclient: monmap: 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:43.000 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 librados: starting msgr at 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 librados: starting objecter 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 Processor -- start 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 -- start start 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 librados: setting wanted keys 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 1 librados: calling monclient init 2024-03-25T18:33:43.001 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.998+0000 7f41b9797640 10 monclient: init 2024-03-25T18:33:43.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:43.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:43.002 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-b con 0x7f41b40a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-c con 0x7f41b413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient(hunting): picked mon.noname-a con 0x7f41b41430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b41d82d0 con 0x7f41b41430c0 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40abb90 con 0x7f41b40a98f0 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b41d76a0 con 0x7f41b413f710 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:43.003 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:42.999+0000 7f41b9797640 10 monclient(hunting): authenticate will time out at 8701.311539s 2024-03-25T18:33:43.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.013+0000 7f41abfff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f41b413f710 0x7f41b41d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:36072/0 (socket says 172.21.15.176:36072) 2024-03-25T18:33:43.014 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.013+0000 7f41abfff640 1 -- 172.21.15.176:0/896603000 learned_addr learned my addr 172.21.15.176:0/896603000 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:43.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.014+0000 7f41ab7fe640 10 client.?.objecter ms_handle_connect 0x7f41b413f710 2024-03-25T18:33:43.015 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.014+0000 7f41ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:43.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.016+0000 7f41ab7fe640 10 client.?.objecter ms_handle_connect 0x7f41b40a98f0 2024-03-25T18:33:43.017 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.016+0000 7f41ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:43.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.017+0000 7f41ab7fe640 10 client.?.objecter ms_handle_connect 0x7f41b41430c0 2024-03-25T18:33:43.018 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.017+0000 7f41ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:43.022 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093588588 0 0) 0x7f41b41d76a0 con 0x7f41b413f710 2024-03-25T18:33:43.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:43.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:43.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:43.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:33:43.023 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.021+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4194004eb0 con 0x7f41b413f710 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018030112 0 0) 0x7f41b40abb90 con 0x7f41b40a98f0 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:33:43.027 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.026+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b41d76a0 con 0x7f41b40a98f0 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022980523 0 0) 0x7f4194004eb0 con 0x7f41b413f710 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b40abb90 con 0x7f41b413f710 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f419c001bc0 con 0x7f41b413f710 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:43.032 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:43.033 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.031+0000 7f41ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:43.034 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867566011 0 0) 0x7f41b41d82d0 con 0x7f41b41430c0 2024-03-25T18:33:43.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:43.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:43.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:43.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:33:43.035 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.033+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4194004eb0 con 0x7f41b41430c0 2024-03-25T18:33:43.052 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189191809 0 0) 0x7f41b40abb90 con 0x7f41b413f710 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 >> v1:172.21.15.186:6789/0 conn(0x7f41b40a98f0 legacy=0x7f41b41385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 >> v1:172.21.15.176:6789/0 conn(0x7f41b41430c0 legacy=0x7f41b4134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 1 monclient: found mon.c 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4134720 con 0x7f41b413f710 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 5 monclient: authenticate success, global_id 4661 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 10 monclient: _renew_subs 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:43.053 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b4134970 con 0x7f41b413f710 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 10 monclient: _renew_subs 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4137a10 con 0x7f41b413f710 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.051+0000 7f41b9797640 1 librados: init done 2024-03-25T18:33:43.054 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.052+0000 7f41b9797640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:43.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.052+0000 7f41b9797640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:43.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.052+0000 7f41b9797640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:43.055 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.052+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b41d7830 con 0x7f41b413f710 2024-03-25T18:33:43.337 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.336+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f419c0011f0 con 0x7f41b413f710 2024-03-25T18:33:43.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.336+0000 7f41ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:43.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.336+0000 7f41b1ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.346+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f419c003b00 con 0x7f41b413f710 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.346+0000 7f41ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.346+0000 7f41ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.346+0000 7f41ab7fe640 10 monclient: dump: 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:43.347 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.347+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f419c003ce0 con 0x7f41b413f710 2024-03-25T18:33:43.348 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.347+0000 7f41b2ffd640 10 monclient: get_auth_request con 0x7f419406de00 auth_method 0 2024-03-25T18:33:43.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.347+0000 7f41ab7fe640 10 client.4661.objecter ms_handle_connect 0x7f419406de00 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f419c054900 con 0x7f41b413f710 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 10 client.4661.objecter ms_dispatch 0x7f41b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T18:33:43.355 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f419c050240 con 0x7f41b413f710 2024-03-25T18:33:43.356 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:43.356 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.354+0000 7f41ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:43.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.604+0000 7f41b9797640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:43.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.604+0000 7f41b9797640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:43.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.604+0000 7f41b9797640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:33:43.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.604+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41b41d7490 con 0x7f41b413f710 2024-03-25T18:33:43.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.633+0000 7f41ab7fe640 1 -- 172.21.15.176:0/896603000 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f419c052340 con 0x7f41b413f710 2024-03-25T18:33:43.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.633+0000 7f41ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:43.635 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.633+0000 7f41ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:43.635 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:43.635 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:43.637 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 10 librados: watch_flush enter 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 10 librados: watch_flush exit 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 10 client.4661.objecter successfully canceled tick 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 >> v1:172.21.15.176:6800/2598213983 conn(0x7f419406de00 legacy=0x7f41940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 10 monclient: shutdown 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 >> v1:172.21.15.176:6790/0 conn(0x7f41b413f710 legacy=0x7f41b41d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 shutdown_connections 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 >> 172.21.15.176:0/896603000 conn(0x7f41b40b7ea0 msgr2=0x7f41b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 shutdown_connections 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.636+0000 7f41b9797640 1 -- 172.21.15.176:0/896603000 wait complete. 2024-03-25T18:33:43.638 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:43.637+0000 7f41b9797640 1 librados: shutdown 2024-03-25T18:33:43.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:50.457 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.572+0000 7f68d2b48640 10 monclient: get_monmap_and_config 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.572+0000 7f68d2b48640 10 monclient: build_initial_monmap 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.572+0000 7f68d2b48640 10 monclient: monmap: 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:50.573788+0000 2024-03-25T18:33:50.574 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:50.573788+0000 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:50.575 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.573+0000 7f68d2b48640 1 Processor -- start 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.573+0000 7f68d2b48640 1 -- start start 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.573+0000 7f68d2b48640 10 monclient: init 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc0b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc0b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc14be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0af940 con 0x7f68cc14be90 2024-03-25T18:33:50.577 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0b0a70 con 0x7f68cc0b1730 2024-03-25T18:33:50.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0b0d50 con 0x7f68cc0b4d60 2024-03-25T18:33:50.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:50.578 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.574+0000 7f68d2b48640 10 monclient(hunting): authenticate will time out at 8708.886651s 2024-03-25T18:33:50.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.581+0000 7f68cbfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f68cc0b1730 0x7f68cc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:58272/0 (socket says 172.21.15.176:58272) 2024-03-25T18:33:50.582 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.581+0000 7f68cbfff640 1 -- 172.21.15.176:0/3911595636 learned_addr learned my addr 172.21.15.176:0/3911595636 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270567576 0 0) 0x7f68cc0b0a70 con 0x7f68cc0b1730 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:33:50.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.591+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc003520 con 0x7f68cc0b1730 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740693159 0 0) 0x7f68bc003520 con 0x7f68cc0b1730 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 >> v1:172.21.15.176:6790/0 conn(0x7f68cc0b4d60 legacy=0x7f68cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 >> v1:172.21.15.176:6789/0 conn(0x7f68cc14be90 legacy=0x7f68cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc0b1030 con 0x7f68cc0b1730 2024-03-25T18:33:50.603 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f68b8001690 con 0x7f68cc0b1730 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68d2b48640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68d2b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68cb7fe640 10 monclient: dump: 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.604 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.605 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.601+0000 7f68d2b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68b8001860 con 0x7f68cc0b1730 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 1 -- 172.21.15.176:0/3911595636 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f68b8001cb0 con 0x7f68cc0b1730 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68d2b48640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68d2b48640 4 set_mon_vals no callback set 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:50.622 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.620+0000 7f68cb7fe640 10 monclient: dump: 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:50.623 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 10 monclient: shutdown 2024-03-25T18:33:50.626 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 1 -- 172.21.15.176:0/3911595636 >> v1:172.21.15.186:6789/0 conn(0x7f68cc0b1730 legacy=0x7f68cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 1 -- 172.21.15.176:0/3911595636 shutdown_connections 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 1 -- 172.21.15.176:0/3911595636 >> 172.21.15.176:0/3911595636 conn(0x7f68cc0617a0 msgr2=0x7f68cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 1 -- 172.21.15.176:0/3911595636 shutdown_connections 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.621+0000 7f68d2b48640 1 -- 172.21.15.176:0/3911595636 wait complete. 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.622+0000 7f68d2b48640 10 monclient: build_initial_monmap 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.622+0000 7f68d2b48640 10 monclient: monmap: 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:50.627 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 librados: starting msgr at 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 librados: starting objecter 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 Processor -- start 2024-03-25T18:33:50.628 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 -- start start 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 librados: setting wanted keys 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 1 librados: calling monclient init 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.623+0000 7f68d2b48640 10 monclient: init 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc0b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc0b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc14be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc1dab20 con 0x7f68cc0b1730 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc1db9a0 con 0x7f68cc0b4d60 2024-03-25T18:33:50.629 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc1dad70 con 0x7f68cc14be90 2024-03-25T18:33:50.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:50.630 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.624+0000 7f68d2b48640 10 monclient(hunting): authenticate will time out at 8708.936552s 2024-03-25T18:33:50.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.630+0000 7f68cbfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f68cc0b4d60 0x7f68cc132e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:58336/0 (socket says 172.21.15.176:58336) 2024-03-25T18:33:50.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.630+0000 7f68cbfff640 1 -- 172.21.15.176:0/1127015247 learned_addr learned my addr 172.21.15.176:0/1127015247 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:50.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.630+0000 7f68c97fa640 10 client.?.objecter ms_handle_connect 0x7f68cc0b4d60 2024-03-25T18:33:50.632 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.631+0000 7f68c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:50.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.633+0000 7f68c97fa640 10 client.?.objecter ms_handle_connect 0x7f68cc0b1730 2024-03-25T18:33:50.634 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.633+0000 7f68c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798198618 0 0) 0x7f68cc1db9a0 con 0x7f68cc0b4d60 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:33:50.642 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.640+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b0004eb0 con 0x7f68cc0b4d60 2024-03-25T18:33:50.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865719585 0 0) 0x7f68cc1dab20 con 0x7f68cc0b1730 2024-03-25T18:33:50.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:50.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:50.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:50.643 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:33:50.644 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.642+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68cc1db9a0 con 0x7f68cc0b1730 2024-03-25T18:33:50.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.644+0000 7f68c97fa640 10 client.?.objecter ms_handle_connect 0x7f68cc14be90 2024-03-25T18:33:50.645 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.644+0000 7f68c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:50.655 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2821329662 0 0) 0x7f68cc1db9a0 con 0x7f68cc0b1730 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68cc1dab20 con 0x7f68cc0b1730 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f68c0001930 con 0x7f68cc0b1730 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 10 monclient(hunting): dump: 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.656 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:50.657 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:50.657 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:50.657 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.654+0000 7f68c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:50.665 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.663+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610154412 0 0) 0x7f68b0004eb0 con 0x7f68cc0b4d60 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.664+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68cc1db9a0 con 0x7f68cc0b4d60 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.664+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f68b8001b70 con 0x7f68cc0b4d60 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.664+0000 7f68c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.664+0000 7f68c97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.664+0000 7f68c97fa640 10 monclient(hunting): dump: 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.666 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:50.667 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.667 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:50.667 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:50.667 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:50.667 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717455951 0 0) 0x7f68cc1dab20 con 0x7f68cc0b1730 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 >> v1:172.21.15.176:6790/0 conn(0x7f68cc14be90 legacy=0x7f68cc139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 >> v1:172.21.15.186:6789/0 conn(0x7f68cc0b4d60 legacy=0x7f68cc132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 1 monclient: found mon.a 2024-03-25T18:33:50.670 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc139300 con 0x7f68cc0b1730 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 10 monclient: _renew_subs 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc133530 con 0x7f68cc0b1730 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:50.671 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 10 monclient: _renew_subs 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc13a770 con 0x7f68cc0b1730 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.668+0000 7f68d2b48640 1 librados: init done 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.669+0000 7f68d2b48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.669+0000 7f68d2b48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.669+0000 7f68d2b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:50.672 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.669+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc1394f0 con 0x7f68cc0b1730 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c0003610 con 0x7f68cc0b1730 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f68c0003960 con 0x7f68cc0b1730 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68c97fa640 10 monclient: dump: 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.893 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:50.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.892+0000 7f68caffd640 4 set_mon_vals no callback set 2024-03-25T18:33:50.980 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.979+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f68c0003ac0 con 0x7f68cc0b1730 2024-03-25T18:33:50.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.980+0000 7f68cbfff640 10 monclient: get_auth_request con 0x7f68b006de80 auth_method 0 2024-03-25T18:33:50.981 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.980+0000 7f68c97fa640 10 client.4635.objecter ms_handle_connect 0x7f68b006de80 2024-03-25T18:33:50.988 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f68c0053680 con 0x7f68cc0b1730 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 10 client.4635.objecter ms_dispatch 0x7f68cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68c004f190 con 0x7f68cc0b1730 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:50.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:50.987+0000 7f68c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:51.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.220+0000 7f68d2b48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:51.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.220+0000 7f68d2b48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:51.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.220+0000 7f68d2b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:51.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.220+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68cc1daf00 con 0x7f68cc0b1730 2024-03-25T18:33:51.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.267+0000 7f68c97fa640 1 -- 172.21.15.176:0/1127015247 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f68c00510c0 con 0x7f68cc0b1730 2024-03-25T18:33:51.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.267+0000 7f68c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:51.268 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.267+0000 7f68c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:51.269 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:51.269 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.269+0000 7f68d2b48640 10 librados: watch_flush enter 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.269+0000 7f68d2b48640 10 librados: watch_flush exit 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.269+0000 7f68d2b48640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.269+0000 7f68d2b48640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 >> v1:172.21.15.176:6800/2598213983 conn(0x7f68b006de80 legacy=0x7f68b0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.271 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 10 monclient: shutdown 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 >> v1:172.21.15.176:6789/0 conn(0x7f68cc0b1730 legacy=0x7f68cc1db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 shutdown_connections 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 >> 172.21.15.176:0/1127015247 conn(0x7f68cc0617a0 msgr2=0x7f68cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 shutdown_connections 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 -- 172.21.15.176:0/1127015247 wait complete. 2024-03-25T18:33:51.272 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:51.270+0000 7f68d2b48640 1 librados: shutdown 2024-03-25T18:33:51.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:58.089 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:58.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.206+0000 7f36fea1e640 10 monclient: get_monmap_and_config 2024-03-25T18:33:58.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.206+0000 7f36fea1e640 10 monclient: build_initial_monmap 2024-03-25T18:33:58.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.206+0000 7f36fea1e640 10 monclient: monmap: 2024-03-25T18:33:58.207 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:33:58.207373+0000 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:33:58.207373+0000 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:33:58.208 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 1 Processor -- start 2024-03-25T18:33:58.209 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 1 -- start start 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient: init 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-a con 0x7f36f80a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-b con 0x7f36f813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-c con 0x7f36f81430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.207+0000 7f36fea1e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f8142520 con 0x7f36f80a98f0 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.208+0000 7f36fea1e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80b28a0 con 0x7f36f81430c0 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.208+0000 7f36fea1e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80b2a30 con 0x7f36f813f710 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.208+0000 7f36fea1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:58.210 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.208+0000 7f36fea1e640 10 monclient(hunting): authenticate will time out at 8716.519867s 2024-03-25T18:33:58.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.219+0000 7f36f7fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f36f80a98f0 0x7f36f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:60766/0 (socket says 172.21.15.176:60766) 2024-03-25T18:33:58.220 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.219+0000 7f36f7fff640 1 -- 172.21.15.176:0/1478962026 learned_addr learned my addr 172.21.15.176:0/1478962026 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:58.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.232+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751572973 0 0) 0x7f36f8142520 con 0x7f36f80a98f0 2024-03-25T18:33:58.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.232+0000 7f36f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.232+0000 7f36f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.232+0000 7f36f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.232+0000 7f36f6ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:33:58.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.233+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e0003520 con 0x7f36f80a98f0 2024-03-25T18:33:58.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304766014 0 0) 0x7f36f80b28a0 con 0x7f36f81430c0 2024-03-25T18:33:58.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:33:58.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.239+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f8142520 con 0x7f36f81430c0 2024-03-25T18:33:58.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901541646 0 0) 0x7f36f80b2a30 con 0x7f36f813f710 2024-03-25T18:33:58.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:33:58.249 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.247+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f80b28a0 con 0x7f36f813f710 2024-03-25T18:33:58.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548908864 0 0) 0x7f36e0003520 con 0x7f36f80a98f0 2024-03-25T18:33:58.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 >> v1:172.21.15.176:6790/0 conn(0x7f36f813f710 legacy=0x7f36f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 >> v1:172.21.15.186:6789/0 conn(0x7f36f81430c0 legacy=0x7f36f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.250 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80b44c0 con 0x7f36f80a98f0 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f36e8001690 con 0x7f36f80a98f0 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36fea1e640 5 monclient: authenticate success, global_id 4638 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36fea1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36f6ffd640 10 monclient: dump: 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:58.251 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.252 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.249+0000 7f36fea1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:58.258 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e8001870 con 0x7f36f80a98f0 2024-03-25T18:33:58.258 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 1 -- 172.21.15.176:0/1478962026 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f36e8001c80 con 0x7f36f80a98f0 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36fea1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36fea1e640 4 set_mon_vals no callback set 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.257+0000 7f36f6ffd640 10 monclient: dump: 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:58.259 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 10 monclient: shutdown 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 1 -- 172.21.15.176:0/1478962026 >> v1:172.21.15.176:6789/0 conn(0x7f36f80a98f0 legacy=0x7f36f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 1 -- 172.21.15.176:0/1478962026 shutdown_connections 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 1 -- 172.21.15.176:0/1478962026 >> 172.21.15.176:0/1478962026 conn(0x7f36f80b7ea0 msgr2=0x7f36f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 1 -- 172.21.15.176:0/1478962026 shutdown_connections 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.258+0000 7f36fea1e640 1 -- 172.21.15.176:0/1478962026 wait complete. 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.259+0000 7f36fea1e640 10 monclient: build_initial_monmap 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.259+0000 7f36fea1e640 10 monclient: monmap: 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:33:58.261 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 librados: starting msgr at 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 librados: starting objecter 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 Processor -- start 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 -- start start 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 librados: setting wanted keys 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 1 librados: calling monclient init 2024-03-25T18:33:58.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.260+0000 7f36fea1e640 10 monclient: init 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-b con 0x7f36f80a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-c con 0x7f36f813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient(hunting): picked mon.noname-a con 0x7f36f81430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f81d51f0 con 0x7f36f81430c0 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f8137990 con 0x7f36f80a98f0 2024-03-25T18:33:58.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f8139990 con 0x7f36f813f710 2024-03-25T18:33:58.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:58.264 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.261+0000 7f36fea1e640 10 monclient(hunting): authenticate will time out at 8716.572998s 2024-03-25T18:33:58.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.272+0000 7f36fcfbb640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f36f81430c0 0x7f36f81db410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:60814/0 (socket says 172.21.15.176:60814) 2024-03-25T18:33:58.273 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.272+0000 7f36fcfbb640 1 -- 172.21.15.176:0/2221828796 learned_addr learned my addr 172.21.15.176:0/2221828796 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:33:58.274 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.273+0000 7f36f4ff9640 10 client.?.objecter ms_handle_connect 0x7f36f81430c0 2024-03-25T18:33:58.274 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.273+0000 7f36f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:58.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.274+0000 7f36f4ff9640 10 client.?.objecter ms_handle_connect 0x7f36f813f710 2024-03-25T18:33:58.275 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.274+0000 7f36f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823414933 0 0) 0x7f36f81d51f0 con 0x7f36f81430c0 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:33:58.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.280+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f36f81430c0 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33993217 0 0) 0x7f36f8139990 con 0x7f36f813f710 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:33:58.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.283+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f81d51f0 con 0x7f36f813f710 2024-03-25T18:33:58.285 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.285+0000 7f36f4ff9640 10 client.?.objecter ms_handle_connect 0x7f36f80a98f0 2024-03-25T18:33:58.286 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.285+0000 7f36f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.295+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273303794 0 0) 0x7f36e4004eb0 con 0x7f36f81430c0 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f8139990 con 0x7f36f81430c0 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f36f0001c00 con 0x7f36f81430c0 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.297 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:58.298 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:58.298 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:58.298 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.298 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.296+0000 7f36f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.301+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341352024 0 0) 0x7f36f8137990 con 0x7f36f80a98f0 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.302+0000 7f36f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.302+0000 7f36f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.302+0000 7f36f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.302+0000 7f36f4ff9640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:33:58.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.302+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f36f80a98f0 2024-03-25T18:33:58.305 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1175711199 0 0) 0x7f36f81d51f0 con 0x7f36f813f710 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f8137990 con 0x7f36f813f710 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f36ec0018b0 con 0x7f36f813f710 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.304+0000 7f36f4ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:58.306 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2844277706 0 0) 0x7f36f8139990 con 0x7f36f81430c0 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 >> v1:172.21.15.176:6790/0 conn(0x7f36f813f710 legacy=0x7f36f81dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 >> v1:172.21.15.186:6789/0 conn(0x7f36f80a98f0 legacy=0x7f36f8137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 1 monclient: found mon.a 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f81385c0 con 0x7f36f81430c0 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:58.311 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 5 monclient: authenticate success, global_id 4641 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 10 monclient: _renew_subs 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f8138810 con 0x7f36f81430c0 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 10 monclient: _renew_subs 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f81e3000 con 0x7f36f81430c0 2024-03-25T18:33:58.312 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.309+0000 7f36fea1e640 1 librados: init done 2024-03-25T18:33:58.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.310+0000 7f36fea1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:58.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.310+0000 7f36fea1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:58.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.310+0000 7f36fea1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.313 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.310+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f80627b0 con 0x7f36f81430c0 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.609+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f00020d0 con 0x7f36f81430c0 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.609+0000 7f36f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.610+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f36f0003b10 con 0x7f36f81430c0 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.610+0000 7f36f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.610+0000 7f36f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.610+0000 7f36f4ff9640 10 monclient: dump: 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:33:58.611 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.610+0000 7f36f67fc640 4 set_mon_vals no callback set 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.611+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f36f0003d30 con 0x7f36f81430c0 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.611+0000 7f36f7fff640 10 monclient: get_auth_request con 0x7f36e406df70 auth_method 0 2024-03-25T18:33:58.612 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.611+0000 7f36f4ff9640 10 client.4641.objecter ms_handle_connect 0x7f36e406df70 2024-03-25T18:33:58.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f36f00549d0 con 0x7f36f81430c0 2024-03-25T18:33:58.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 10 client.4641.objecter ms_dispatch 0x7f36f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:58.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 3 client.4641.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:58.617 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 3 client.4641.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:58.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 20 client.4641.objecter dump_active .. 0 homeless 2024-03-25T18:33:58.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36f0050310 con 0x7f36f81430c0 2024-03-25T18:33:58.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:58.618 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.616+0000 7f36f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:58.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.870+0000 7f36fea1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:58.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.870+0000 7f36fea1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:58.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.870+0000 7f36fea1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:33:58.871 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.870+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36f8137b20 con 0x7f36f81430c0 2024-03-25T18:33:58.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.889+0000 7f36f4ff9640 1 -- 172.21.15.176:0/2221828796 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f36f0052410 con 0x7f36f81430c0 2024-03-25T18:33:58.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.889+0000 7f36f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:58.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.889+0000 7f36f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:58.891 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:33:58.891 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.892+0000 7f36fea1e640 10 librados: watch_flush enter 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.892+0000 7f36fea1e640 10 librados: watch_flush exit 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.892+0000 7f36fea1e640 20 client.4641.objecter shutdown clearing up homeless session... 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.892+0000 7f36fea1e640 10 client.4641.objecter successfully canceled tick 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 >> v1:172.21.15.176:6800/2598213983 conn(0x7f36e406df70 legacy=0x7f36e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 10 monclient: shutdown 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 >> v1:172.21.15.176:6789/0 conn(0x7f36f81430c0 legacy=0x7f36f81db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:58.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 shutdown_connections 2024-03-25T18:33:58.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 >> 172.21.15.176:0/2221828796 conn(0x7f36f80b7ea0 msgr2=0x7f36f813c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:58.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 shutdown_connections 2024-03-25T18:33:58.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 -- 172.21.15.176:0/2221828796 wait complete. 2024-03-25T18:33:58.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:33:58.893+0000 7f36fea1e640 1 librados: shutdown 2024-03-25T18:33:58.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:59.269 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:01.025 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.314 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:01.048 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.316 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:01.051 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.317 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:01.073 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.318 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:01.074 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.320 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:01.076 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.345 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:01.101 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.359 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:01.093 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:59.370 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:01.126 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:05.714 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:05.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.829+0000 7f2428c0f640 10 monclient: get_monmap_and_config 2024-03-25T18:34:05.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.829+0000 7f2428c0f640 10 monclient: build_initial_monmap 2024-03-25T18:34:05.830 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.829+0000 7f2428c0f640 10 monclient: monmap: 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:05.830232+0000 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:05.830232+0000 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:05.831 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.830+0000 7f2428c0f640 1 Processor -- start 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.830+0000 7f2428c0f640 1 -- start start 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.830+0000 7f2428c0f640 10 monclient: init 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:05.832 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f24240a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f242413f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f24241430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2424142520 con 0x7f242413f710 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240b28a0 con 0x7f24241430c0 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240b2a30 con 0x7f24240a98f0 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:05.833 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.831+0000 7f2428c0f640 10 monclient(hunting): authenticate will time out at 8724.142960s 2024-03-25T18:34:05.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.843+0000 7f2422d9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f24241430c0 0x7f242414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:38882/0 (socket says 172.21.15.176:38882) 2024-03-25T18:34:05.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.843+0000 7f2422d9d640 1 -- 172.21.15.176:0/1612842880 learned_addr learned my addr 172.21.15.176:0/1612842880 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:05.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904295703 0 0) 0x7f24240b28a0 con 0x7f24241430c0 2024-03-25T18:34:05.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:34:05.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.855+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2410003520 con 0x7f24241430c0 2024-03-25T18:34:05.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537743195 0 0) 0x7f24240b2a30 con 0x7f24240a98f0 2024-03-25T18:34:05.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:34:05.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.860+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24240b28a0 con 0x7f24240a98f0 2024-03-25T18:34:05.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057093696 0 0) 0x7f2424142520 con 0x7f242413f710 2024-03-25T18:34:05.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.863 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:34:05.864 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.862+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24240b2a30 con 0x7f242413f710 2024-03-25T18:34:05.884 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2198791036 0 0) 0x7f2410003520 con 0x7f24241430c0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 >> v1:172.21.15.176:6790/0 conn(0x7f24240a98f0 legacy=0x7f24240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 >> v1:172.21.15.176:6789/0 conn(0x7f242413f710 legacy=0x7f24241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 monclient: found mon.noname-c 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240b44c0 con 0x7f24241430c0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f241c001a10 con 0x7f24241430c0 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:05.885 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2421d9b640 10 monclient: dump: 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2428c0f640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:34:05.886 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.883+0000 7f2428c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:05.896 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f241c001b70 con 0x7f24241430c0 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 1 -- 172.21.15.176:0/1612842880 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f241c001f80 con 0x7f24241430c0 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2421d9b640 10 monclient: dump: 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.897 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2428c0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:05.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.895+0000 7f2428c0f640 4 set_mon_vals no callback set 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 10 monclient: shutdown 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 1 -- 172.21.15.176:0/1612842880 >> v1:172.21.15.186:6789/0 conn(0x7f24241430c0 legacy=0x7f242414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 1 -- 172.21.15.176:0/1612842880 shutdown_connections 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 1 -- 172.21.15.176:0/1612842880 >> 172.21.15.176:0/1612842880 conn(0x7f24240b7ea0 msgr2=0x7f24240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.896+0000 7f2428c0f640 1 -- 172.21.15.176:0/1612842880 shutdown_connections 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.897+0000 7f2428c0f640 1 -- 172.21.15.176:0/1612842880 wait complete. 2024-03-25T18:34:05.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 10 monclient: build_initial_monmap 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 10 monclient: monmap: 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 librados: starting msgr at 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 librados: starting objecter 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 Processor -- start 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 -- start start 2024-03-25T18:34:05.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 librados: setting wanted keys 2024-03-25T18:34:05.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 1 librados: calling monclient init 2024-03-25T18:34:05.901 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.898+0000 7f2428c0f640 10 monclient: init 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f24240a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f242413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f24241430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2424135230 con 0x7f24240a98f0 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2424135da0 con 0x7f242413f710 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2424135f30 con 0x7f24241430c0 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:05.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.899+0000 7f2428c0f640 10 monclient(hunting): authenticate will time out at 8724.211579s 2024-03-25T18:34:05.912 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.911+0000 7f2422d9d640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f24241430c0 0x7f24241db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:49524/0 (socket says 172.21.15.176:49524) 2024-03-25T18:34:05.912 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.911+0000 7f2422d9d640 1 -- 172.21.15.176:0/1558105357 learned_addr learned my addr 172.21.15.176:0/1558105357 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:05.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.915+0000 7f241affd640 10 client.?.objecter ms_handle_connect 0x7f24241430c0 2024-03-25T18:34:05.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.915+0000 7f241affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:05.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.917+0000 7f241affd640 10 client.?.objecter ms_handle_connect 0x7f24240a98f0 2024-03-25T18:34:05.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.917+0000 7f241affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:05.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.919+0000 7f241affd640 10 client.?.objecter ms_handle_connect 0x7f242413f710 2024-03-25T18:34:05.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.919+0000 7f241affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:05.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208318090 0 0) 0x7f2424135da0 con 0x7f242413f710 2024-03-25T18:34:05.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.932 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2404004eb0 con 0x7f242413f710 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178197621 0 0) 0x7f2424135f30 con 0x7f24241430c0 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.931+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424135da0 con 0x7f24241430c0 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723882443 0 0) 0x7f2424135230 con 0x7f24240a98f0 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:05.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:05.934 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:05.934 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:34:05.934 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.932+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424135f30 con 0x7f24240a98f0 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 685802141 0 0) 0x7f2404004eb0 con 0x7f242413f710 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2424135230 con 0x7f242413f710 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f240c0018f0 con 0x7f242413f710 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 10 monclient(hunting): dump: 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:05.941 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:05.942 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:05.942 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:05.942 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:05.942 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.942 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.940+0000 7f241affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.947+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225231688 0 0) 0x7f2424135da0 con 0x7f24241430c0 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.948+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2404004eb0 con 0x7f24241430c0 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.948+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f241c001e80 con 0x7f24241430c0 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.948+0000 7f241affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.948+0000 7f241affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.948+0000 7f241affd640 10 monclient(hunting): dump: 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:05.949 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3747826708 0 0) 0x7f2424135230 con 0x7f242413f710 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 >> v1:172.21.15.176:6790/0 conn(0x7f24241430c0 legacy=0x7f24241db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 >> v1:172.21.15.176:6789/0 conn(0x7f24240a98f0 legacy=0x7f24241d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 1 monclient: found mon.b 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24241360c0 con 0x7f242413f710 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 10 monclient: _finish_auth 0 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:05.954 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f241affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 10 monclient: _renew_subs 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24241d5000 con 0x7f242413f710 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 10 monclient: _renew_subs 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24241d41e0 con 0x7f242413f710 2024-03-25T18:34:05.955 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.952+0000 7f2428c0f640 1 librados: init done 2024-03-25T18:34:05.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.953+0000 7f2428c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:05.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.953+0000 7f2428c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:05.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.953+0000 7f2428c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:05.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:05.953+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2424135090 con 0x7f242413f710 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f240c003610 con 0x7f242413f710 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f240c003780 con 0x7f242413f710 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f241affd640 10 monclient: dump: 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:06.171 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:06.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.170+0000 7f242159a640 4 set_mon_vals no callback set 2024-03-25T18:34:06.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.228+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f240c019750 con 0x7f242413f710 2024-03-25T18:34:06.229 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.228+0000 7f242259c640 10 monclient: get_auth_request con 0x7f240406df70 auth_method 0 2024-03-25T18:34:06.230 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.229+0000 7f241affd640 10 client.4669.objecter ms_handle_connect 0x7f240406df70 2024-03-25T18:34:06.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.231+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f240c053450 con 0x7f242413f710 2024-03-25T18:34:06.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.231+0000 7f241affd640 10 client.4669.objecter ms_dispatch 0x7f2424000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:06.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.231+0000 7f241affd640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:06.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.231+0000 7f241affd640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:06.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.231+0000 7f241affd640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:34:06.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.233+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f240c04ed90 con 0x7f242413f710 2024-03-25T18:34:06.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.233+0000 7f241affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:06.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.233+0000 7f241affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:06.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.494+0000 7f2428c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:06.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.494+0000 7f2428c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:06.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.494+0000 7f2428c0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:06.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.494+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24241353c0 con 0x7f242413f710 2024-03-25T18:34:06.532 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.531+0000 7f241affd640 1 -- 172.21.15.176:0/1558105357 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f240c050e90 con 0x7f242413f710 2024-03-25T18:34:06.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.531+0000 7f241affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:06.533 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.531+0000 7f241affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:06.533 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:06.533 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 10 librados: watch_flush enter 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 10 librados: watch_flush exit 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 >> v1:172.21.15.176:6800/2598213983 conn(0x7f240406df70 legacy=0x7f2404070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 10 monclient: shutdown 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 >> v1:172.21.15.186:6789/0 conn(0x7f242413f710 legacy=0x7f24241dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 shutdown_connections 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 >> 172.21.15.176:0/1558105357 conn(0x7f24240b7ea0 msgr2=0x7f242414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:06.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 shutdown_connections 2024-03-25T18:34:06.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 -- 172.21.15.176:0/1558105357 wait complete. 2024-03-25T18:34:06.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:06.534+0000 7f2428c0f640 1 librados: shutdown 2024-03-25T18:34:06.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:13.355 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.470+0000 7fa08ac25640 10 monclient: get_monmap_and_config 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.471+0000 7fa08ac25640 10 monclient: build_initial_monmap 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.471+0000 7fa08ac25640 10 monclient: monmap: 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:13.471866+0000 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:13.471866+0000 2024-03-25T18:34:13.472 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:13.473 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.473 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:13.473 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:13.473 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:13.473 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.471+0000 7fa08ac25640 1 Processor -- start 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.471+0000 7fa08ac25640 1 -- start start 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.471+0000 7fa08ac25640 10 monclient: init 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-c con 0x7fa0840a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:13.474 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-b con 0x7fa08413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-a con 0x7fa0841430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa084142520 con 0x7fa0841430c0 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840b28a0 con 0x7fa0840a98f0 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840b2a30 con 0x7fa08413f710 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:13.475 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.472+0000 7fa08ac25640 10 monclient(hunting): authenticate will time out at 8731.784363s 2024-03-25T18:34:13.480 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.479+0000 7fa0891c2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa0841430c0 0x7fa08414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:60132/0 (socket says 172.21.15.176:60132) 2024-03-25T18:34:13.481 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.479+0000 7fa0891c2640 1 -- 172.21.15.176:0/2154401003 learned_addr learned my addr 172.21.15.176:0/2154401003 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56658372 0 0) 0x7fa084142520 con 0x7fa0841430c0 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:34:13.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.494+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa074003520 con 0x7fa0841430c0 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423825137 0 0) 0x7fa0840b28a0 con 0x7fa0840a98f0 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.507+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084142520 con 0x7fa0840a98f0 2024-03-25T18:34:13.509 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230033357 0 0) 0x7fa074003520 con 0x7fa0841430c0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 >> v1:172.21.15.176:6790/0 conn(0x7fa08413f710 legacy=0x7fa0841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 >> v1:172.21.15.186:6789/0 conn(0x7fa0840a98f0 legacy=0x7fa0840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 monclient: found mon.noname-a 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0840b44c0 con 0x7fa0841430c0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa07c001a10 con 0x7fa0841430c0 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:13.510 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa0837fe640 10 monclient: dump: 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa08ac25640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:34:13.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.508+0000 7fa08ac25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c001b70 con 0x7fa0841430c0 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 1 -- 172.21.15.176:0/2154401003 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa07c001f80 con 0x7fa0841430c0 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:13.514 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa0837fe640 10 monclient: dump: 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa08ac25640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:13.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.513+0000 7fa08ac25640 4 set_mon_vals no callback set 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 10 monclient: shutdown 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 1 -- 172.21.15.176:0/2154401003 >> v1:172.21.15.176:6789/0 conn(0x7fa0841430c0 legacy=0x7fa08414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 1 -- 172.21.15.176:0/2154401003 shutdown_connections 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 1 -- 172.21.15.176:0/2154401003 >> 172.21.15.176:0/2154401003 conn(0x7fa0840b7ea0 msgr2=0x7fa0840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 1 -- 172.21.15.176:0/2154401003 shutdown_connections 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.514+0000 7fa08ac25640 1 -- 172.21.15.176:0/2154401003 wait complete. 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 10 monclient: build_initial_monmap 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 10 monclient: monmap: 2024-03-25T18:34:13.517 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 librados: starting msgr at 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 librados: starting objecter 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 Processor -- start 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 -- start start 2024-03-25T18:34:13.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 librados: setting wanted keys 2024-03-25T18:34:13.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 1 librados: calling monclient init 2024-03-25T18:34:13.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.515+0000 7fa08ac25640 10 monclient: init 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-a con 0x7fa0840a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-c con 0x7fa08413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient(hunting): picked mon.noname-b con 0x7fa0841430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa084136c50 con 0x7fa0840a98f0 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa084136ec0 con 0x7fa0841430c0 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0841317f0 con 0x7fa08413f710 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:13.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.516+0000 7fa08ac25640 10 monclient(hunting): authenticate will time out at 8731.828746s 2024-03-25T18:34:13.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.534+0000 7fa083fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa08413f710 0x7fa0841dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:54264/0 (socket says 172.21.15.176:54264) 2024-03-25T18:34:13.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.534+0000 7fa083fff640 1 -- 172.21.15.176:0/3168581666 learned_addr learned my addr 172.21.15.176:0/3168581666 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:13.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.535+0000 7fa0817fa640 10 client.?.objecter ms_handle_connect 0x7fa08413f710 2024-03-25T18:34:13.536 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.535+0000 7fa0817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:13.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.540+0000 7fa0817fa640 10 client.?.objecter ms_handle_connect 0x7fa0840a98f0 2024-03-25T18:34:13.541 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.540+0000 7fa0817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:13.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.546+0000 7fa0817fa640 10 client.?.objecter ms_handle_connect 0x7fa0841430c0 2024-03-25T18:34:13.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.546+0000 7fa0817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:13.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47880105 0 0) 0x7fa084136c50 con 0x7fa0840a98f0 2024-03-25T18:34:13.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:13.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:13.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:13.548 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:34:13.549 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.547+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068004eb0 con 0x7fa0840a98f0 2024-03-25T18:34:13.554 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361417410 0 0) 0x7fa0841317f0 con 0x7fa08413f710 2024-03-25T18:34:13.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:13.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:13.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:13.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:34:13.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.553+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084136c50 con 0x7fa08413f710 2024-03-25T18:34:13.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551437145 0 0) 0x7fa084136ec0 con 0x7fa0841430c0 2024-03-25T18:34:13.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:13.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:13.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:13.566 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:34:13.567 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.565+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0841317f0 con 0x7fa0841430c0 2024-03-25T18:34:13.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181607798 0 0) 0x7fa068004eb0 con 0x7fa0840a98f0 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa084136ec0 con 0x7fa0840a98f0 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa078001970 con 0x7fa0840a98f0 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 10 monclient(hunting): dump: 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.573 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:13.574 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:13.574 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:13.574 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.571+0000 7fa0817fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:13.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3956154825 0 0) 0x7fa084136c50 con 0x7fa08413f710 2024-03-25T18:34:13.579 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa068004eb0 con 0x7fa08413f710 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa070001970 con 0x7fa08413f710 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.578+0000 7fa0817fa640 10 monclient(hunting): dump: 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:13.580 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1574487038 0 0) 0x7fa084136ec0 con 0x7fa0840a98f0 2024-03-25T18:34:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 >> v1:172.21.15.176:6790/0 conn(0x7fa08413f710 legacy=0x7fa0841dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 >> v1:172.21.15.186:6789/0 conn(0x7fa0841430c0 legacy=0x7fa0841db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:13.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 1 monclient: found mon.a 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa084131980 con 0x7fa0840a98f0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa0817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 10 monclient: _renew_subs 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa084135070 con 0x7fa0840a98f0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 10 monclient: _renew_subs 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0840b7870 con 0x7fa0840a98f0 2024-03-25T18:34:13.592 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.590+0000 7fa08ac25640 1 librados: init done 2024-03-25T18:34:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.591+0000 7fa08ac25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.591+0000 7fa08ac25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.591+0000 7fa08ac25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:13.593 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.591+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0840627b0 con 0x7fa0840a98f0 2024-03-25T18:34:13.914 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa078003610 con 0x7fa0840a98f0 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa078003920 con 0x7fa0840a98f0 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa0817fa640 10 monclient: dump: 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:13.915 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.913+0000 7fa082ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.914+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa0780198f0 con 0x7fa0840a98f0 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.915+0000 7fa083fff640 10 monclient: get_auth_request con 0x7fa06806df20 auth_method 0 2024-03-25T18:34:13.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.915+0000 7fa0817fa640 10 client.4653.objecter ms_handle_connect 0x7fa06806df20 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7fa0780536b0 con 0x7fa0840a98f0 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 10 client.4653.objecter ms_dispatch 0x7fa084000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa07804eff0 con 0x7fa0840a98f0 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:13.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:13.921+0000 7fa0817fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:14.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.156+0000 7fa08ac25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:14.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.156+0000 7fa08ac25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:14.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.156+0000 7fa08ac25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:14.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.156+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa08413df70 con 0x7fa0840a98f0 2024-03-25T18:34:14.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.188+0000 7fa0817fa640 1 -- 172.21.15.176:0/3168581666 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7fa0780510f0 con 0x7fa0840a98f0 2024-03-25T18:34:14.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.188+0000 7fa0817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:14.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.188+0000 7fa0817fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:14.189 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:14.190 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:14.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.190+0000 7fa08ac25640 10 librados: watch_flush enter 2024-03-25T18:34:14.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.190+0000 7fa08ac25640 10 librados: watch_flush exit 2024-03-25T18:34:14.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.190+0000 7fa08ac25640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:34:14.192 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.190+0000 7fa08ac25640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa06806df20 legacy=0x7fa0680703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 10 monclient: shutdown 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 >> v1:172.21.15.176:6789/0 conn(0x7fa0840a98f0 legacy=0x7fa0841310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 shutdown_connections 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 >> 172.21.15.176:0/3168581666 conn(0x7fa0840b7ea0 msgr2=0x7fa08413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 shutdown_connections 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 -- 172.21.15.176:0/3168581666 wait complete. 2024-03-25T18:34:14.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:14.191+0000 7fa08ac25640 1 librados: shutdown 2024-03-25T18:34:14.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:19.399 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:21.155 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.406 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:21.162 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.443 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:21.199 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.466 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:21.200 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.473 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:21.208 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.474 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:21.230 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.484 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:21.240 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.504 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:21.238 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:19.526 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:21.282 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:21.012 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:21.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.125+0000 7f4e13f79640 10 monclient: get_monmap_and_config 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.125+0000 7f4e13f79640 10 monclient: build_initial_monmap 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.125+0000 7f4e13f79640 10 monclient: monmap: 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:21.126459+0000 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:21.126459+0000 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:21.127 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.126+0000 7f4e13f79640 1 Processor -- start 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.126+0000 7f4e13f79640 1 -- start start 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.126+0000 7f4e13f79640 10 monclient: init 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c0a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c1430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c142520 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b28a0 con 0x7f4e0c13f710 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b2a30 con 0x7f4e0c1430c0 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:21.129 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.127+0000 7f4e13f79640 10 monclient(hunting): authenticate will time out at 8739.439494s 2024-03-25T18:34:21.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.140+0000 7f4e11d15640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4e0c0a98f0 0x7f4e0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54294/0 (socket says 172.21.15.176:54294) 2024-03-25T18:34:21.141 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.140+0000 7f4e11d15640 1 -- 172.21.15.176:0/4286743738 learned_addr learned my addr 172.21.15.176:0/4286743738 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691826855 0 0) 0x7f4e0c0b2a30 con 0x7f4e0c1430c0 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:34:21.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.153+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df8003520 con 0x7f4e0c1430c0 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496532197 0 0) 0x7f4e0c0b28a0 con 0x7f4e0c13f710 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:34:21.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.154+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0b2a30 con 0x7f4e0c13f710 2024-03-25T18:34:21.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283187735 0 0) 0x7f4e0c142520 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:34:21.158 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.156+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0b28a0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.161 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894461522 0 0) 0x7f4df8003520 con 0x7f4e0c1430c0 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 >> v1:172.21.15.186:6789/0 conn(0x7f4e0c13f710 legacy=0x7f4e0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 >> v1:172.21.15.176:6789/0 conn(0x7f4e0c0a98f0 legacy=0x7f4e0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 10 monclient: _finish_hunting 0 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 1 monclient: found mon.noname-b 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0b44c0 con 0x7f4e0c1430c0 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 10 monclient: _finish_auth 0 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 10 monclient: _check_auth_tickets 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e10d13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e13f79640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:34:21.162 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.160+0000 7f4e13f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4e040019d0 con 0x7f4e0c1430c0 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: dump: 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.174 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e04001c20 con 0x7f4e0c1430c0 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 1 -- 172.21.15.176:0/4286743738 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4e04001fb0 con 0x7f4e0c1430c0 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 4 set_mon_vals no callback set 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:21.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e10d13640 10 monclient: dump: 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:21.176 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.178 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 10 monclient: shutdown 2024-03-25T18:34:21.178 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.173+0000 7f4e13f79640 1 -- 172.21.15.176:0/4286743738 >> v1:172.21.15.176:6790/0 conn(0x7f4e0c1430c0 legacy=0x7f4e0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 1 -- 172.21.15.176:0/4286743738 shutdown_connections 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 1 -- 172.21.15.176:0/4286743738 >> 172.21.15.176:0/4286743738 conn(0x7f4e0c0b7ea0 msgr2=0x7f4e0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 1 -- 172.21.15.176:0/4286743738 shutdown_connections 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 1 -- 172.21.15.176:0/4286743738 wait complete. 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 10 monclient: build_initial_monmap 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.174+0000 7f4e13f79640 10 monclient: monmap: 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:21.179 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 librados: starting msgr at 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 librados: starting objecter 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 Processor -- start 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 -- start start 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 librados: setting wanted keys 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 1 librados: calling monclient init 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient: init 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c13f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.175+0000 7f4e13f79640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.176+0000 7f4e13f79640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c12fca0 con 0x7f4e0c1430c0 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.176+0000 7f4e13f79640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c137470 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.176+0000 7f4e13f79640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c137750 con 0x7f4e0c13f710 2024-03-25T18:34:21.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.176+0000 7f4e13f79640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:21.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.176+0000 7f4e13f79640 10 monclient(hunting): authenticate will time out at 8739.487906s 2024-03-25T18:34:21.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.188+0000 7f4e11d15640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4e0c0a98f0 0x7f4e0c138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:57596/0 (socket says 172.21.15.176:57596) 2024-03-25T18:34:21.189 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.188+0000 7f4e11d15640 1 -- 172.21.15.176:0/112131157 learned_addr learned my addr 172.21.15.176:0/112131157 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:21.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.192+0000 7f4e027fc640 10 client.?.objecter ms_handle_connect 0x7f4e0c0a98f0 2024-03-25T18:34:21.193 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.192+0000 7f4e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:21.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.196+0000 7f4e027fc640 10 client.?.objecter ms_handle_connect 0x7f4e0c1430c0 2024-03-25T18:34:21.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.196+0000 7f4e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:21.198 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.197+0000 7f4e027fc640 10 client.?.objecter ms_handle_connect 0x7f4e0c13f710 2024-03-25T18:34:21.198 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.197+0000 7f4e027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:21.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020551718 0 0) 0x7f4e0c137470 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.205 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:34:21.206 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.204+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.207 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564531199 0 0) 0x7f4e0c137750 con 0x7f4e0c13f710 2024-03-25T18:34:21.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:34:21.208 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.207+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c137470 con 0x7f4e0c13f710 2024-03-25T18:34:21.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615276045 0 0) 0x7f4e0c12fca0 con 0x7f4e0c1430c0 2024-03-25T18:34:21.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:21.221 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:21.222 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:21.222 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:34:21.222 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.220+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c137750 con 0x7f4e0c1430c0 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.221+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775624622 0 0) 0x7f4df0004eb0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e0c12fca0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4e08001910 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 10 monclient(hunting): dump: 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.223 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.222+0000 7f4e027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088367865 0 0) 0x7f4e0c137470 con 0x7f4e0c13f710 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e0c13f710 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4dfc0018b0 con 0x7f4e0c13f710 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:21.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.225+0000 7f4e027fc640 10 monclient(hunting): dump: 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:21.227 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3654268736 0 0) 0x7f4e0c12fca0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.232 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 >> v1:172.21.15.176:6790/0 conn(0x7f4e0c13f710 legacy=0x7f4e0c130070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 >> v1:172.21.15.176:6789/0 conn(0x7f4e0c1430c0 legacy=0x7f4e0c130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 1 monclient: found mon.b 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c136520 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e13f79640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e13f79640 10 monclient: _renew_subs 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e13f79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.230+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c1353a0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:21.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 monclient: _renew_subs 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c135e60 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 1 librados: init done 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:21.234 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.231+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c0627b0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e080047e0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f4e08004bf0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e027fc640 10 monclient: dump: 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:21.511 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.509+0000 7f4e03fff640 4 set_mon_vals no callback set 2024-03-25T18:34:21.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.510+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f4e08004e10 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.511+0000 7f4e11514640 10 monclient: get_auth_request con 0x7f4df006df50 auth_method 0 2024-03-25T18:34:21.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.511+0000 7f4e027fc640 10 client.4681.objecter ms_handle_connect 0x7f4df006df50 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f4e08054890 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 10 client.4681.objecter ms_dispatch 0x7f4e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:34:21.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e080503a0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:21.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.516+0000 7f4e027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:21.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.753+0000 7f4e13f79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:21.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.753+0000 7f4e13f79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:21.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.753+0000 7f4e13f79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:21.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.753+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e0c141a00 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.771+0000 7f4e027fc640 1 -- 172.21.15.176:0/112131157 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f4e080522d0 con 0x7f4e0c0a98f0 2024-03-25T18:34:21.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.771+0000 7f4e027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:21.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.771+0000 7f4e027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:21.773 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:21.773 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:21.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 10 librados: watch_flush enter 2024-03-25T18:34:21.775 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 10 librados: watch_flush exit 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 >> v1:172.21.15.176:6800/2598213983 conn(0x7f4df006df50 legacy=0x7f4df0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 10 monclient: shutdown 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 >> v1:172.21.15.186:6789/0 conn(0x7f4e0c0a98f0 legacy=0x7f4e0c138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 shutdown_connections 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 >> 172.21.15.176:0/112131157 conn(0x7f4e0c0b7ea0 msgr2=0x7f4e0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 shutdown_connections 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 -- 172.21.15.176:0/112131157 wait complete. 2024-03-25T18:34:21.776 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:21.774+0000 7f4e13f79640 1 librados: shutdown 2024-03-25T18:34:21.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:28.595 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:28.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.709+0000 7f96c5a0f640 10 monclient: get_monmap_and_config 2024-03-25T18:34:28.710 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.709+0000 7f96c5a0f640 10 monclient: build_initial_monmap 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.709+0000 7f96c5a0f640 10 monclient: monmap: 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:28.710518+0000 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:28.710518+0000 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:28.711 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.710+0000 7f96c5a0f640 1 Processor -- start 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.710+0000 7f96c5a0f640 1 -- start start 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.710+0000 7f96c5a0f640 10 monclient: init 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f96c0061830 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f96c00709d0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f96c013c8c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c0071cd0 con 0x7f96c00709d0 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c0071fb0 con 0x7f96c013c8c0 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c006ff10 con 0x7f96c0061830 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:28.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.711+0000 7f96c5a0f640 10 monclient(hunting): authenticate will time out at 8747.023702s 2024-03-25T18:34:28.715 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.715+0000 7f96beffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f96c0061830 0x7f96c013c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:60178/0 (socket says 172.21.15.176:60178) 2024-03-25T18:34:28.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.715+0000 7f96beffd640 1 -- 172.21.15.176:0/3224917530 learned_addr learned my addr 172.21.15.176:0/3224917530 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:28.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437060745 0 0) 0x7f96c006ff10 con 0x7f96c0061830 2024-03-25T18:34:28.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:28.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:28.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:28.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:34:28.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.719+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac005730 con 0x7f96c0061830 2024-03-25T18:34:28.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299421266 0 0) 0x7f96ac005730 con 0x7f96c0061830 2024-03-25T18:34:28.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 >> v1:172.21.15.186:6789/0 conn(0x7f96c013c8c0 legacy=0x7f96c0072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:28.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 >> v1:172.21.15.176:6789/0 conn(0x7f96c00709d0 legacy=0x7f96c00723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:28.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 monclient: found mon.noname-b 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c014e950 con 0x7f96c0061830 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f96b0001670 con 0x7f96c0061830 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96c5a0f640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96c5a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96be7fc640 10 monclient: dump: 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:28.728 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:28.729 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.726+0000 7f96c5a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:28.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b0001840 con 0x7f96c0061830 2024-03-25T18:34:28.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:28.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 1 -- 172.21.15.176:0/3224917530 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f96b0001c50 con 0x7f96c0061830 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96c5a0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96c5a0f640 4 set_mon_vals no callback set 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.745+0000 7f96be7fc640 10 monclient: dump: 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:28.747 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:28.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 10 monclient: shutdown 2024-03-25T18:34:28.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:28.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 1 -- 172.21.15.176:0/3224917530 >> v1:172.21.15.176:6790/0 conn(0x7f96c0061830 legacy=0x7f96c013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:28.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 1 -- 172.21.15.176:0/3224917530 shutdown_connections 2024-03-25T18:34:28.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 1 -- 172.21.15.176:0/3224917530 >> 172.21.15.176:0/3224917530 conn(0x7f96c00b11e0 msgr2=0x7f96c00b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 1 -- 172.21.15.176:0/3224917530 shutdown_connections 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.746+0000 7f96c5a0f640 1 -- 172.21.15.176:0/3224917530 wait complete. 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 10 monclient: build_initial_monmap 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 10 monclient: monmap: 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 librados: starting msgr at 2024-03-25T18:34:28.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 librados: starting objecter 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 Processor -- start 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 -- start start 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 librados: setting wanted keys 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 1 librados: calling monclient init 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.747+0000 7f96c5a0f640 10 monclient: init 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f96c0061830 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f96c00709d0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f96c013c8c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00ac390 con 0x7f96c013c8c0 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c01363e0 con 0x7f96c00709d0 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00ad480 con 0x7f96c0061830 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:28.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.748+0000 7f96c5a0f640 10 monclient(hunting): authenticate will time out at 8747.060348s 2024-03-25T18:34:28.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.757+0000 7f96b7fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f96c00709d0 0x7f96c00acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:37376/0 (socket says 172.21.15.176:37376) 2024-03-25T18:34:28.758 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.757+0000 7f96b7fff640 1 -- 172.21.15.176:0/2139757418 learned_addr learned my addr 172.21.15.176:0/2139757418 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:28.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.760+0000 7f96bcff9640 10 client.?.objecter ms_handle_connect 0x7f96c00709d0 2024-03-25T18:34:28.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.760+0000 7f96bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:28.762 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.761+0000 7f96bcff9640 10 client.?.objecter ms_handle_connect 0x7f96c013c8c0 2024-03-25T18:34:28.762 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.761+0000 7f96bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:28.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.769+0000 7f96bcff9640 10 client.?.objecter ms_handle_connect 0x7f96c0061830 2024-03-25T18:34:28.769 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.769+0000 7f96bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151668478 0 0) 0x7f96c01363e0 con 0x7f96c00709d0 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:34:28.772 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.771+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0004eb0 con 0x7f96c00709d0 2024-03-25T18:34:28.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970903900 0 0) 0x7f96c00ac390 con 0x7f96c013c8c0 2024-03-25T18:34:28.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:28.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:28.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:28.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:34:28.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.777+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c01363e0 con 0x7f96c013c8c0 2024-03-25T18:34:28.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369584887 0 0) 0x7f96a0004eb0 con 0x7f96c00709d0 2024-03-25T18:34:28.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96c00ac390 con 0x7f96c00709d0 2024-03-25T18:34:28.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f96a8001930 con 0x7f96c00709d0 2024-03-25T18:34:28.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:28.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 10 monclient(hunting): dump: 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.786+0000 7f96bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:28.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46616815 0 0) 0x7f96c00ad480 con 0x7f96c0061830 2024-03-25T18:34:28.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:28.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:28.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:28.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:34:28.789 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.787+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0004eb0 con 0x7f96c0061830 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 407864071 0 0) 0x7f96c00ac390 con 0x7f96c00709d0 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 >> v1:172.21.15.176:6790/0 conn(0x7f96c0061830 legacy=0x7f96c00a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 >> v1:172.21.15.176:6789/0 conn(0x7f96c013c8c0 legacy=0x7f96c00ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 1 monclient: found mon.b 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c012bac0 con 0x7f96c00709d0 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:34:28.797 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 10 monclient: _renew_subs 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c00aa7f0 con 0x7f96c00709d0 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 10 monclient: _renew_subs 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c00ae250 con 0x7f96c00709d0 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.795+0000 7f96c5a0f640 1 librados: init done 2024-03-25T18:34:28.798 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.796+0000 7f96c5a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:28.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.796+0000 7f96c5a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:28.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.796+0000 7f96c5a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:28.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:28.796+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c00627b0 con 0x7f96c00709d0 2024-03-25T18:34:29.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.050+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96a80035f0 con 0x7f96c00709d0 2024-03-25T18:34:29.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.050+0000 7f96bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:29.051 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.050+0000 7f96bdffb640 4 set_mon_vals no callback set 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.072+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f96a8003980 con 0x7f96c00709d0 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.072+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.072+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.072+0000 7f96bcff9640 10 monclient: dump: 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:29.073 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:29.074 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:29.074 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:29.074 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:29.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.073+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f96a8003b60 con 0x7f96c00709d0 2024-03-25T18:34:29.074 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.073+0000 7f96beffd640 10 monclient: get_auth_request con 0x7f96a006ddb0 auth_method 0 2024-03-25T18:34:29.075 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.074+0000 7f96bcff9640 10 client.4684.objecter ms_handle_connect 0x7f96a006ddb0 2024-03-25T18:34:29.079 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.078+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3903806985 0 0) 0x7f96a8054820 con 0x7f96c00709d0 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.078+0000 7f96bcff9640 10 client.4684.objecter ms_dispatch 0x7f96c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.078+0000 7f96bcff9640 3 client.4684.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.078+0000 7f96bcff9640 3 client.4684.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.078+0000 7f96bcff9640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.079+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96a8050330 con 0x7f96c00709d0 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.079+0000 7f96bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:29.080 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.079+0000 7f96bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:29.308 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.307+0000 7f96c5a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:29.308 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.307+0000 7f96c5a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:29.308 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.307+0000 7f96c5a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:29.308 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.307+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96c0137ed0 con 0x7f96c00709d0 2024-03-25T18:34:29.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.326+0000 7f96bcff9640 1 -- 172.21.15.176:0/2139757418 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917114213) 0x7f96a8052260 con 0x7f96c00709d0 2024-03-25T18:34:29.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.326+0000 7f96bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:29.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.326+0000 7f96bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:29.328 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:29.328 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":10,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:24:30.942596+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:30.942596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.328+0000 7f96c5a0f640 10 librados: watch_flush enter 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.328+0000 7f96c5a0f640 10 librados: watch_flush exit 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.328+0000 7f96c5a0f640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 10 client.4684.objecter successfully canceled tick 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 >> v1:172.21.15.176:6800/2598213983 conn(0x7f96a006ddb0 legacy=0x7f96a0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 10 monclient: shutdown 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 >> v1:172.21.15.186:6789/0 conn(0x7f96c00709d0 legacy=0x7f96c00acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 shutdown_connections 2024-03-25T18:34:29.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 >> 172.21.15.176:0/2139757418 conn(0x7f96c00b11e0 msgr2=0x7f96c00af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:29.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 shutdown_connections 2024-03-25T18:34:29.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 -- 172.21.15.176:0/2139757418 wait complete. 2024-03-25T18:34:29.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:29.329+0000 7f96c5a0f640 1 librados: shutdown 2024-03-25T18:34:29.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:36.148 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:36.262 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.261+0000 7fa6de577640 10 monclient: get_monmap_and_config 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.261+0000 7fa6de577640 10 monclient: build_initial_monmap 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.261+0000 7fa6de577640 10 monclient: monmap: 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:36.262785+0000 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:36.262785+0000 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:36.263 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.262+0000 7fa6de577640 1 Processor -- start 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.262+0000 7fa6de577640 1 -- start start 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.262+0000 7fa6de577640 10 monclient: init 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d80b4d60 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d80b1730 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d814be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80af940 con 0x7fa6d80b4d60 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80b0a70 con 0x7fa6d80b1730 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80b0d50 con 0x7fa6d814be90 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:36.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.263+0000 7fa6de577640 10 monclient(hunting): authenticate will time out at 8754.575651s 2024-03-25T18:34:36.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.269+0000 7fa6dcb14640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa6d814be90 0x7fa6d814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53744/0 (socket says 172.21.15.176:53744) 2024-03-25T18:34:36.270 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.269+0000 7fa6dcb14640 1 -- 172.21.15.176:0/561127481 learned_addr learned my addr 172.21.15.176:0/561127481 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:36.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404773845 0 0) 0x7fa6d80b0a70 con 0x7fa6d80b1730 2024-03-25T18:34:36.281 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:36.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:36.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:36.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:34:36.282 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.280+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0003520 con 0x7fa6d80b1730 2024-03-25T18:34:36.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128648610 0 0) 0x7fa6d80b0d50 con 0x7fa6d814be90 2024-03-25T18:34:36.283 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:36.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:36.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:36.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:34:36.284 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.282+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d80b0a70 con 0x7fa6d814be90 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 831676580 0 0) 0x7fa6c0003520 con 0x7fa6d80b1730 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 >> v1:172.21.15.176:6790/0 conn(0x7fa6d814be90 legacy=0x7fa6d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 >> v1:172.21.15.176:6789/0 conn(0x7fa6d80b4d60 legacy=0x7fa6d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:36.302 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80b1030 con 0x7fa6d80b1730 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa6cc001690 con 0x7fa6d80b1730 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6de577640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6de577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6d6ffd640 10 monclient: dump: 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:36.303 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.304 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:36.304 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:36.304 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:36.304 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.304 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.301+0000 7fa6de577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6cc0017f0 con 0x7fa6d80b1730 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 1 -- 172.21.15.176:0/561127481 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa6cc001c00 con 0x7fa6d80b1730 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6d6ffd640 10 monclient: dump: 2024-03-25T18:34:36.316 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6de577640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:36.317 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.315+0000 7fa6de577640 4 set_mon_vals no callback set 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 10 monclient: shutdown 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 1 -- 172.21.15.176:0/561127481 >> v1:172.21.15.186:6789/0 conn(0x7fa6d80b1730 legacy=0x7fa6d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 1 -- 172.21.15.176:0/561127481 shutdown_connections 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 1 -- 172.21.15.176:0/561127481 >> 172.21.15.176:0/561127481 conn(0x7fa6d80617a0 msgr2=0x7fa6d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 1 -- 172.21.15.176:0/561127481 shutdown_connections 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.316+0000 7fa6de577640 1 -- 172.21.15.176:0/561127481 wait complete. 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.317+0000 7fa6de577640 10 monclient: build_initial_monmap 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.317+0000 7fa6de577640 10 monclient: monmap: 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:36.319 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.317+0000 7fa6de577640 1 librados: starting msgr at 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.317+0000 7fa6de577640 1 librados: starting objecter 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.317+0000 7fa6de577640 1 Processor -- start 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 1 -- start start 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 1 librados: setting wanted keys 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 1 librados: calling monclient init 2024-03-25T18:34:36.320 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 10 monclient: init 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d80b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.318+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d80b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d814be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:36.321 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8142050 con 0x7fa6d80b1730 2024-03-25T18:34:36.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d81dac50 con 0x7fa6d814be90 2024-03-25T18:34:36.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8140d00 con 0x7fa6d80b4d60 2024-03-25T18:34:36.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:36.322 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.319+0000 7fa6de577640 10 monclient(hunting): authenticate will time out at 8754.630955s 2024-03-25T18:34:36.326 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.325+0000 7fa6d77fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa6d80b4d60 0x7fa6d81db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:53790/0 (socket says 172.21.15.176:53790) 2024-03-25T18:34:36.327 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.326+0000 7fa6d77fe640 1 -- 172.21.15.176:0/3879808081 learned_addr learned my addr 172.21.15.176:0/3879808081 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:36.329 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.329+0000 7fa6d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6d80b4d60 2024-03-25T18:34:36.330 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.329+0000 7fa6d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:36.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.330+0000 7fa6d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6d814be90 2024-03-25T18:34:36.331 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.330+0000 7fa6d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:36.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.332+0000 7fa6d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6d80b1730 2024-03-25T18:34:36.332 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.332+0000 7fa6d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100737109 0 0) 0x7fa6d8142050 con 0x7fa6d80b1730 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:34:36.338 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.337+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4004eb0 con 0x7fa6d80b1730 2024-03-25T18:34:36.342 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416819239 0 0) 0x7fa6d81dac50 con 0x7fa6d814be90 2024-03-25T18:34:36.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:36.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:36.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:36.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:34:36.343 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.342+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8142050 con 0x7fa6d814be90 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642162269 0 0) 0x7fa6d8142050 con 0x7fa6d814be90 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d81dac50 con 0x7fa6d814be90 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa6d0001b50 con 0x7fa6d814be90 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.349 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.350 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.348+0000 7fa6d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:36.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578313165 0 0) 0x7fa6c4004eb0 con 0x7fa6d80b1730 2024-03-25T18:34:36.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d8142050 con 0x7fa6d80b1730 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa6c8001930 con 0x7fa6d80b1730 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:36.352 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587824801 0 0) 0x7fa6d8140d00 con 0x7fa6d80b4d60 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:34:36.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.350+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4004eb0 con 0x7fa6d80b4d60 2024-03-25T18:34:36.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1872009732 0 0) 0x7fa6d81dac50 con 0x7fa6d814be90 2024-03-25T18:34:36.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 >> v1:172.21.15.176:6790/0 conn(0x7fa6d80b4d60 legacy=0x7fa6d81db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 >> v1:172.21.15.176:6789/0 conn(0x7fa6d80b1730 legacy=0x7fa6d81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.368 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 1 monclient: found mon.b 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d81423e0 con 0x7fa6d814be90 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 10 monclient: _renew_subs 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d813b470 con 0x7fa6d814be90 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 10 monclient: _renew_subs 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.369 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d81dba40 con 0x7fa6d814be90 2024-03-25T18:34:36.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.367+0000 7fa6de577640 1 librados: init done 2024-03-25T18:34:36.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.368+0000 7fa6de577640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:36.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.368+0000 7fa6de577640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:36.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.368+0000 7fa6de577640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.368+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d81401f0 con 0x7fa6d814be90 2024-03-25T18:34:36.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d0002000 con 0x7fa6d814be90 2024-03-25T18:34:36.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:36.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa6d0004d00 con 0x7fa6d814be90 2024-03-25T18:34:36.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:36.650 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d4ff9640 10 monclient: dump: 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:36.651 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.649+0000 7fa6d67fc640 4 set_mon_vals no callback set 2024-03-25T18:34:36.652 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.650+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa6d001acd0 con 0x7fa6d814be90 2024-03-25T18:34:36.652 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.650+0000 7fa6d7fff640 10 monclient: get_auth_request con 0x7fa6c406df50 auth_method 0 2024-03-25T18:34:36.652 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.651+0000 7fa6d4ff9640 10 client.4690.objecter ms_handle_connect 0x7fa6c406df50 2024-03-25T18:34:36.657 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fa6d00548d0 con 0x7fa6d814be90 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 10 client.4690.objecter ms_dispatch 0x7fa6d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6d0050210 con 0x7fa6d814be90 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:36.658 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.656+0000 7fa6d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:36.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.893+0000 7fa6de577640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:36.894 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.893+0000 7fa6de577640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:36.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.893+0000 7fa6de577640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:36.895 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.893+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6d8140e90 con 0x7fa6d814be90 2024-03-25T18:34:36.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.932+0000 7fa6d4ff9640 1 -- 172.21.15.176:0/3879808081 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fa6d0052310 con 0x7fa6d814be90 2024-03-25T18:34:36.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.932+0000 7fa6d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:36.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.932+0000 7fa6d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:36.933 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:36.934 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:36.936 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 10 librados: watch_flush enter 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 10 librados: watch_flush exit 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 10 client.4690.objecter successfully canceled tick 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa6c406df50 legacy=0x7fa6c4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 10 monclient: shutdown 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 >> v1:172.21.15.186:6789/0 conn(0x7fa6d814be90 legacy=0x7fa6d81e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 shutdown_connections 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 >> 172.21.15.176:0/3879808081 conn(0x7fa6d80617a0 msgr2=0x7fa6d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 shutdown_connections 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 -- 172.21.15.176:0/3879808081 wait complete. 2024-03-25T18:34:36.937 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:36.935+0000 7fa6de577640 1 librados: shutdown 2024-03-25T18:34:36.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:39.425 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:41.181 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.426 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:41.183 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.457 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:10:41.213 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.483 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:41.217 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.485 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:41.219 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.494 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:41.250 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.502 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:41.258 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.519 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:41.253 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:39.543 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:10:41.299 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:43.755 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.869+0000 7f62aabc8640 10 monclient: get_monmap_and_config 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.869+0000 7f62aabc8640 10 monclient: build_initial_monmap 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.869+0000 7f62aabc8640 10 monclient: monmap: 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:43.870074+0000 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:43.870074+0000 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:43.870 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:43.871 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:43.871 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:43.871 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:43.871 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 1 Processor -- start 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 1 -- start start 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient: init 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-c con 0x7f62a40b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:43.872 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-a con 0x7f62a40b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-b con 0x7f62a414be90 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40af940 con 0x7f62a40b1730 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40b0a70 con 0x7f62a40b4d60 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40b0d50 con 0x7f62a414be90 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:43.873 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.870+0000 7f62aabc8640 10 monclient(hunting): authenticate will time out at 8762.182827s 2024-03-25T18:34:43.882 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.882+0000 7f62a3fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f62a40b1730 0x7f62a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:47182/0 (socket says 172.21.15.176:47182) 2024-03-25T18:34:43.883 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.882+0000 7f62a3fff640 1 -- 172.21.15.176:0/2201809542 learned_addr learned my addr 172.21.15.176:0/2201809542 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281808313 0 0) 0x7f62a40b0a70 con 0x7f62a40b4d60 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:34:43.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.899+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6294003520 con 0x7f62a40b4d60 2024-03-25T18:34:43.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053821582 0 0) 0x7f62a40af940 con 0x7f62a40b1730 2024-03-25T18:34:43.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:43.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:43.902 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:43.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:34:43.903 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.901+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a40b0a70 con 0x7f62a40b1730 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355232198 0 0) 0x7f6294003520 con 0x7f62a40b4d60 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 >> v1:172.21.15.176:6790/0 conn(0x7f62a414be90 legacy=0x7f62a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 >> v1:172.21.15.176:6789/0 conn(0x7f62a40b1730 legacy=0x7f62a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 monclient: found mon.noname-c 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40b1030 con 0x7f62a40b4d60 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:43.916 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6298001690 con 0x7f62a40b4d60 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62aabc8640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62aabc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.914+0000 7f62a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.915+0000 7f62a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.915+0000 7f62a37fe640 10 monclient: dump: 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:43.917 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:43.918 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.915+0000 7f62aabc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62980017f0 con 0x7f62a40b4d60 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 1 -- 172.21.15.176:0/2201809542 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6298001c00 con 0x7f62a40b4d60 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62aabc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62aabc8640 4 set_mon_vals no callback set 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.923+0000 7f62a37fe640 10 monclient: dump: 2024-03-25T18:34:43.924 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:43.925 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 10 monclient: shutdown 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 1 -- 172.21.15.176:0/2201809542 >> v1:172.21.15.186:6789/0 conn(0x7f62a40b4d60 legacy=0x7f62a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 1 -- 172.21.15.176:0/2201809542 shutdown_connections 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 1 -- 172.21.15.176:0/2201809542 >> 172.21.15.176:0/2201809542 conn(0x7f62a40617a0 msgr2=0x7f62a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:43.926 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 1 -- 172.21.15.176:0/2201809542 shutdown_connections 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.924+0000 7f62aabc8640 1 -- 172.21.15.176:0/2201809542 wait complete. 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 10 monclient: build_initial_monmap 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 10 monclient: monmap: 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 librados: starting msgr at 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 librados: starting objecter 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 Processor -- start 2024-03-25T18:34:43.927 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 -- start start 2024-03-25T18:34:43.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 librados: setting wanted keys 2024-03-25T18:34:43.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 1 librados: calling monclient init 2024-03-25T18:34:43.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.925+0000 7f62aabc8640 10 monclient: init 2024-03-25T18:34:43.928 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-c con 0x7f62a40b1730 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-b con 0x7f62a40b4d60 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient(hunting): picked mon.noname-a con 0x7f62a414be90 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a4137600 con 0x7f62a414be90 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a4144320 con 0x7f62a40b4d60 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a4144600 con 0x7f62a40b1730 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.926+0000 7f62aabc8640 10 monclient(hunting): authenticate will time out at 8762.238681s 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.927+0000 7f62a3fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f62a40b4d60 0x7f62a4144a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:60822/0 (socket says 172.21.15.176:60822) 2024-03-25T18:34:43.929 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.927+0000 7f62a3fff640 1 -- 172.21.15.176:0/275291478 learned_addr learned my addr 172.21.15.176:0/275291478 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:43.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.929+0000 7f62a17fa640 10 client.?.objecter ms_handle_connect 0x7f62a40b4d60 2024-03-25T18:34:43.930 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.929+0000 7f62a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:43.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.932+0000 7f62a17fa640 10 client.?.objecter ms_handle_connect 0x7f62a40b1730 2024-03-25T18:34:43.933 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.932+0000 7f62a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:43.938 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089302801 0 0) 0x7f62a4144320 con 0x7f62a40b4d60 2024-03-25T18:34:43.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:43.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:43.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:43.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:34:43.939 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.937+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6288004eb0 con 0x7f62a40b4d60 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283599636 0 0) 0x7f6288004eb0 con 0x7f62a40b4d60 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62a4144320 con 0x7f62a40b4d60 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 client.?.objecter ms_handle_connect 0x7f62a414be90 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6290001a30 con 0x7f62a40b4d60 2024-03-25T18:34:43.947 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 monclient(hunting): dump: 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:43.948 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.946+0000 7f62a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774968061 0 0) 0x7f62a4144600 con 0x7f62a40b1730 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:34:43.951 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.950+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6288004eb0 con 0x7f62a40b1730 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2637753114 0 0) 0x7f62a4144320 con 0x7f62a40b4d60 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 >> v1:172.21.15.176:6790/0 conn(0x7f62a40b1730 legacy=0x7f62a4136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 >> v1:172.21.15.176:6789/0 conn(0x7f62a414be90 legacy=0x7f62a4145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 1 monclient: found mon.b 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.954+0000 7f62a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:43.956 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a41367f0 con 0x7f62a40b4d60 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 10 monclient: _renew_subs 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a4136aa0 con 0x7f62a40b4d60 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 10 monclient: _renew_subs 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a41e37c0 con 0x7f62a40b4d60 2024-03-25T18:34:43.957 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.955+0000 7f62aabc8640 1 librados: init done 2024-03-25T18:34:43.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.956+0000 7f62aabc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:43.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.956+0000 7f62aabc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:43.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.956+0000 7f62aabc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:43.958 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:43.956+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a40627b0 con 0x7f62a40b4d60 2024-03-25T18:34:44.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.170+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6290001d50 con 0x7f62a40b4d60 2024-03-25T18:34:44.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.170+0000 7f62a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:44.171 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.170+0000 7f62a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:44.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.236+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6290004c30 con 0x7f62a40b4d60 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.236+0000 7f62a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.236+0000 7f62a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.236+0000 7f62a17fa640 10 monclient: dump: 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:44.237 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:44.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.237+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f629001ac00 con 0x7f62a40b4d60 2024-03-25T18:34:44.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.237+0000 7f62a8964640 10 monclient: get_auth_request con 0x7f628806dd10 auth_method 0 2024-03-25T18:34:44.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.237+0000 7f62a17fa640 10 client.4696.objecter ms_handle_connect 0x7f628806dd10 2024-03-25T18:34:44.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.240+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7f62900549a0 con 0x7f62a40b4d60 2024-03-25T18:34:44.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.240+0000 7f62a17fa640 10 client.4696.objecter ms_dispatch 0x7f62a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:34:44.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.240+0000 7f62a17fa640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:34:44.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.240+0000 7f62a17fa640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:34:44.241 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.240+0000 7f62a17fa640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T18:34:44.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.243+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62900502e0 con 0x7f62a40b4d60 2024-03-25T18:34:44.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.243+0000 7f62a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:44.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.243+0000 7f62a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:44.495 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.494+0000 7f62aabc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:44.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.494+0000 7f62aabc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:44.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.494+0000 7f62aabc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:34:44.496 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.494+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62a4069f40 con 0x7f62a40b4d60 2024-03-25T18:34:44.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.533+0000 7f62a17fa640 1 -- 172.21.15.176:0/275291478 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7f62900523e0 con 0x7f62a40b4d60 2024-03-25T18:34:44.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.533+0000 7f62a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:44.535 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.533+0000 7f62a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:44.535 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:44.535 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:44.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.535+0000 7f62aabc8640 10 librados: watch_flush enter 2024-03-25T18:34:44.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 10 librados: watch_flush exit 2024-03-25T18:34:44.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 10 client.4696.objecter successfully canceled tick 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 >> v1:172.21.15.176:6800/2598213983 conn(0x7f628806dd10 legacy=0x7f62880701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 10 monclient: shutdown 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 >> v1:172.21.15.186:6789/0 conn(0x7f62a40b4d60 legacy=0x7f62a4144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 shutdown_connections 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 >> 172.21.15.176:0/275291478 conn(0x7f62a40617a0 msgr2=0x7f62a40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 shutdown_connections 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 -- 172.21.15.176:0/275291478 wait complete. 2024-03-25T18:34:44.538 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:44.536+0000 7f62aabc8640 1 librados: shutdown 2024-03-25T18:34:44.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:51.358 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.475+0000 7f6e2a7b6640 10 monclient: get_monmap_and_config 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.475+0000 7f6e2a7b6640 10 monclient: build_initial_monmap 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.475+0000 7f6e2a7b6640 10 monclient: monmap: 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:51.476243+0000 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:51.476243+0000 2024-03-25T18:34:51.476 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:51.477 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:51.477 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:51.477 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:51.477 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:51.477 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 1 Processor -- start 2024-03-25T18:34:51.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 1 -- start start 2024-03-25T18:34:51.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient: init 2024-03-25T18:34:51.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:51.478 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e240a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e241430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24142520 con 0x7f6e241430c0 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240b28a0 con 0x7f6e240a98f0 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240b2a30 con 0x7f6e2413f710 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:51.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.476+0000 7f6e2a7b6640 10 monclient(hunting): authenticate will time out at 8769.788740s 2024-03-25T18:34:51.483 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.483+0000 7f6e23fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6e240a98f0 0x7f6e240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:38756/0 (socket says 172.21.15.176:38756) 2024-03-25T18:34:51.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.483+0000 7f6e23fff640 1 -- 172.21.15.176:0/646217284 learned_addr learned my addr 172.21.15.176:0/646217284 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:51.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678253862 0 0) 0x7f6e240b28a0 con 0x7f6e240a98f0 2024-03-25T18:34:51.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.497 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:34:51.498 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.496+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c003520 con 0x7f6e240a98f0 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333954310 0 0) 0x7f6e24142520 con 0x7f6e241430c0 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:34:51.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.502+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e240b28a0 con 0x7f6e241430c0 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834091413 0 0) 0x7f6e240b2a30 con 0x7f6e2413f710 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:34:51.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.511+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e24142520 con 0x7f6e2413f710 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.513+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2935421215 0 0) 0x7f6e0c003520 con 0x7f6e240a98f0 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 >> v1:172.21.15.176:6790/0 conn(0x7f6e2413f710 legacy=0x7f6e241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 >> v1:172.21.15.176:6789/0 conn(0x7f6e241430c0 legacy=0x7f6e2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 1 monclient: found mon.noname-c 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:34:51.515 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240b44c0 con 0x7f6e240a98f0 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6e14001690 con 0x7f6e240a98f0 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e22ffd640 10 monclient: dump: 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:51.516 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:51.517 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e2a7b6640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:34:51.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.514+0000 7f6e2a7b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e140017f0 con 0x7f6e240a98f0 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 1 -- 172.21.15.176:0/646217284 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6e14001c00 con 0x7f6e240a98f0 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e22ffd640 10 monclient: dump: 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.542 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e2a7b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:51.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.541+0000 7f6e2a7b6640 4 set_mon_vals no callback set 2024-03-25T18:34:51.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 10 monclient: shutdown 2024-03-25T18:34:51.544 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/646217284 >> v1:172.21.15.186:6789/0 conn(0x7f6e240a98f0 legacy=0x7f6e240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/646217284 shutdown_connections 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/646217284 >> 172.21.15.176:0/646217284 conn(0x7f6e240b7ea0 msgr2=0x7f6e240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/646217284 shutdown_connections 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.542+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/646217284 wait complete. 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.543+0000 7f6e2a7b6640 10 monclient: build_initial_monmap 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.543+0000 7f6e2a7b6640 10 monclient: monmap: 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:51.545 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.543+0000 7f6e2a7b6640 1 librados: starting msgr at 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.543+0000 7f6e2a7b6640 1 librados: starting objecter 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.543+0000 7f6e2a7b6640 1 Processor -- start 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 -- start start 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 librados: setting wanted keys 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 librados: calling monclient init 2024-03-25T18:34:51.546 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient: init 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e240a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e241430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24135730 con 0x7f6e241430c0 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e241358c0 con 0x7f6e240a98f0 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24135a50 con 0x7f6e2413f710 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:51.547 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.544+0000 7f6e2a7b6640 10 monclient(hunting): authenticate will time out at 8769.856492s 2024-03-25T18:34:51.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.557+0000 7f6e23fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6e240a98f0 0x7f6e241dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:38802/0 (socket says 172.21.15.176:38802) 2024-03-25T18:34:51.557 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.557+0000 7f6e23fff640 1 -- 172.21.15.176:0/1231815196 learned_addr learned my addr 172.21.15.176:0/1231815196 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:51.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.559+0000 7f6e20ff9640 10 client.?.objecter ms_handle_connect 0x7f6e240a98f0 2024-03-25T18:34:51.560 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.559+0000 7f6e20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:51.561 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.561+0000 7f6e20ff9640 10 client.?.objecter ms_handle_connect 0x7f6e241430c0 2024-03-25T18:34:51.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.561+0000 7f6e20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:51.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.564+0000 7f6e20ff9640 10 client.?.objecter ms_handle_connect 0x7f6e2413f710 2024-03-25T18:34:51.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.564+0000 7f6e20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215277591 0 0) 0x7f6e24135730 con 0x7f6e241430c0 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:34:51.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.571+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10004eb0 con 0x7f6e241430c0 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268909248 0 0) 0x7f6e241358c0 con 0x7f6e240a98f0 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:34:51.574 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.573+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e24135730 con 0x7f6e240a98f0 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379364166 0 0) 0x7f6e24135a50 con 0x7f6e2413f710 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:34:51.580 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.579+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e241358c0 con 0x7f6e2413f710 2024-03-25T18:34:51.589 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.588+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957234853 0 0) 0x7f6e10004eb0 con 0x7f6e241430c0 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e24135a50 con 0x7f6e241430c0 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6e1c001bd0 con 0x7f6e241430c0 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:51.590 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.591 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.589+0000 7f6e20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974582198 0 0) 0x7f6e24135a50 con 0x7f6e241430c0 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 >> v1:172.21.15.176:6790/0 conn(0x7f6e2413f710 legacy=0x7f6e241d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 >> v1:172.21.15.186:6789/0 conn(0x7f6e240a98f0 legacy=0x7f6e241dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 1 monclient: found mon.a 2024-03-25T18:34:51.596 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e241d5000 con 0x7f6e241430c0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 10 monclient: _renew_subs 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e241d52b0 con 0x7f6e241430c0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 10 monclient: _renew_subs 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2413c900 con 0x7f6e241430c0 2024-03-25T18:34:51.597 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.595+0000 7f6e2a7b6640 1 librados: init done 2024-03-25T18:34:51.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.596+0000 7f6e2a7b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:51.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.596+0000 7f6e2a7b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:51.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.596+0000 7f6e2a7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:51.598 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.596+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2413cbb0 con 0x7f6e241430c0 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e1c0020e0 con 0x7f6e241430c0 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f6e1c003970 con 0x7f6e241430c0 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e20ff9640 10 monclient: dump: 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:51.898 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:51.899 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.897+0000 7f6e227fc640 4 set_mon_vals no callback set 2024-03-25T18:34:51.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.898+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f6e1c0199c0 con 0x7f6e241430c0 2024-03-25T18:34:51.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.898+0000 7f6e23fff640 10 monclient: get_auth_request con 0x7f6e1006ddf0 auth_method 0 2024-03-25T18:34:51.900 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.899+0000 7f6e20ff9640 10 client.4680.objecter ms_handle_connect 0x7f6e1006ddf0 2024-03-25T18:34:51.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7f6e1c0537a0 con 0x7f6e241430c0 2024-03-25T18:34:51.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 10 client.4680.objecter ms_dispatch 0x7f6e24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 3 client.4680.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 3 client.4680.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 20 client.4680.objecter dump_active .. 0 homeless 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e1c04f0e0 con 0x7f6e241430c0 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:51.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:51.906+0000 7f6e20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:52.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.158+0000 7f6e2a7b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:52.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.158+0000 7f6e2a7b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:52.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.158+0000 7f6e2a7b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:52.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.158+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e241371c0 con 0x7f6e241430c0 2024-03-25T18:34:52.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.232+0000 7f6e20ff9640 1 -- 172.21.15.176:0/1231815196 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7f6e1c0511e0 con 0x7f6e241430c0 2024-03-25T18:34:52.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.232+0000 7f6e20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:52.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.232+0000 7f6e20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:52.233 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:52.233 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 10 librados: watch_flush enter 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 10 librados: watch_flush exit 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 20 client.4680.objecter shutdown clearing up homeless session... 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 10 client.4680.objecter successfully canceled tick 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 >> v1:172.21.15.176:6800/2598213983 conn(0x7f6e1006ddf0 legacy=0x7f6e100702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 10 monclient: shutdown 2024-03-25T18:34:52.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 >> v1:172.21.15.176:6789/0 conn(0x7f6e241430c0 legacy=0x7f6e241d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 shutdown_connections 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.234+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 >> 172.21.15.176:0/1231815196 conn(0x7f6e240b7ea0 msgr2=0x7f6e2414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.235+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 shutdown_connections 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.235+0000 7f6e2a7b6640 1 -- 172.21.15.176:0/1231815196 wait complete. 2024-03-25T18:34:52.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:52.235+0000 7f6e2a7b6640 1 librados: shutdown 2024-03-25T18:34:52.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:59.053 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:59.170 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.169+0000 7f054efa3640 10 monclient: get_monmap_and_config 2024-03-25T18:34:59.170 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.169+0000 7f054efa3640 10 monclient: build_initial_monmap 2024-03-25T18:34:59.170 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.169+0000 7f054efa3640 10 monclient: monmap: 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:34:59.170580+0000 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:34:59.170580+0000 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:34:59.171 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.170+0000 7f054efa3640 1 Processor -- start 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.170+0000 7f054efa3640 1 -- start start 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.170+0000 7f054efa3640 10 monclient: init 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-b con 0x7f054806ab80 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-a con 0x7f054806af60 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-c con 0x7f05480b00b0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:59.172 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480b0500 con 0x7f054806af60 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054813ffa0 con 0x7f05480b00b0 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480af520 con 0x7f054806ab80 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.171+0000 7f054efa3640 10 monclient(hunting): authenticate will time out at 8777.483313s 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.172+0000 7f054cd3f640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f054806ab80 0x7f05480ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:46862/0 (socket says 172.21.15.176:46862) 2024-03-25T18:34:59.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.172+0000 7f054cd3f640 1 -- 172.21.15.176:0/2090890339 learned_addr learned my addr 172.21.15.176:0/2090890339 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:59.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627915357 0 0) 0x7f05480af520 con 0x7f054806ab80 2024-03-25T18:34:59.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:59.186 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:59.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:59.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:34:59.187 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.185+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0538003520 con 0x7f054806ab80 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508826945 0 0) 0x7f054813ffa0 con 0x7f05480b00b0 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:34:59.191 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.190+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05480af520 con 0x7f05480b00b0 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2590793764 0 0) 0x7f0538003520 con 0x7f054806ab80 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 >> v1:172.21.15.186:6789/0 conn(0x7f05480b00b0 legacy=0x7f054814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 >> v1:172.21.15.176:6789/0 conn(0x7f054806af60 legacy=0x7f054813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 1 monclient: found mon.noname-b 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0548072890 con 0x7f054806ab80 2024-03-25T18:34:59.196 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:59.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:59.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f05477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:59.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f054efa3640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:34:59.197 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.195+0000 7f054efa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:59.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.201+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f053c001650 con 0x7f054806ab80 2024-03-25T18:34:59.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.201+0000 7f05477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:59.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.201+0000 7f05477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:59.202 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.201+0000 7f05477fe640 10 monclient: dump: 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.203 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.201+0000 7f054efa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f053c0018a0 con 0x7f054806ab80 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 1 -- 172.21.15.176:0/2090890339 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f053c001650 con 0x7f054806ab80 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f05477fe640 10 monclient: dump: 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.211 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f054efa3640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:59.212 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.210+0000 7f054efa3640 4 set_mon_vals no callback set 2024-03-25T18:34:59.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 10 monclient: shutdown 2024-03-25T18:34:59.213 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 1 -- 172.21.15.176:0/2090890339 >> v1:172.21.15.176:6790/0 conn(0x7f054806ab80 legacy=0x7f05480ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 1 -- 172.21.15.176:0/2090890339 shutdown_connections 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 1 -- 172.21.15.176:0/2090890339 >> 172.21.15.176:0/2090890339 conn(0x7f05480b4a80 msgr2=0x7f05480b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 1 -- 172.21.15.176:0/2090890339 shutdown_connections 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.211+0000 7f054efa3640 1 -- 172.21.15.176:0/2090890339 wait complete. 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 10 monclient: build_initial_monmap 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 10 monclient: monmap: 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:34:59.214 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 librados: starting msgr at 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 librados: starting objecter 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 Processor -- start 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 -- start start 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 librados: setting wanted keys 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 1 librados: calling monclient init 2024-03-25T18:34:59.215 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.212+0000 7f054efa3640 10 monclient: init 2024-03-25T18:34:59.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:59.216 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-a con 0x7f054806ab80 addr v1:172.21.15.176:6789/0 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-c con 0x7f05480b00b0 addr v1:172.21.15.176:6790/0 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient(hunting): picked mon.noname-b con 0x7f05481d7cd0 addr v1:172.21.15.186:6789/0 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05481448a0 con 0x7f054806ab80 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05481454f0 con 0x7f05481d7cd0 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0548146220 con 0x7f05480b00b0 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:59.217 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.213+0000 7f054efa3640 10 monclient(hunting): authenticate will time out at 8777.525821s 2024-03-25T18:34:59.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.223+0000 7f0547fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7f05480b00b0 0x7f05481d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:46910/0 (socket says 172.21.15.176:46910) 2024-03-25T18:34:59.224 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.223+0000 7f0547fff640 1 -- 172.21.15.176:0/91315893 learned_addr learned my addr 172.21.15.176:0/91315893 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:34:59.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.225+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f05480b00b0 2024-03-25T18:34:59.226 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.225+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:59.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.232+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f05481d7cd0 2024-03-25T18:34:59.233 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.232+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686918958 0 0) 0x7f05481454f0 con 0x7f05481d7cd0 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:34:59.240 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.239+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c004eb0 con 0x7f05481d7cd0 2024-03-25T18:34:59.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.241+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f054806ab80 2024-03-25T18:34:59.242 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.241+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184434947 0 0) 0x7f0548146220 con 0x7f05480b00b0 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:34:59.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.243+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05481454f0 con 0x7f05480b00b0 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.246+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937392497 0 0) 0x7f05481448a0 con 0x7f054806ab80 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.246+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.246+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.246+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.247+0000 7f05457fa640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:34:59.248 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.247+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0548146220 con 0x7f054806ab80 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539432132 0 0) 0x7f0548146220 con 0x7f054806ab80 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05481448a0 con 0x7f054806ab80 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f053c001bf0 con 0x7f054806ab80 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 10 monclient(hunting): dump: 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.253 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.254 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.252+0000 7f05457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:59.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291474594 0 0) 0x7f052c004eb0 con 0x7f05481d7cd0 2024-03-25T18:34:59.260 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0548146220 con 0x7f05481d7cd0 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f0540001bd0 con 0x7f05481d7cd0 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.259+0000 7f05457fa640 10 monclient(hunting): dump: 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:59.261 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1958621457 0 0) 0x7f05481448a0 con 0x7f054806ab80 2024-03-25T18:34:59.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 >> v1:172.21.15.176:6790/0 conn(0x7f05480b00b0 legacy=0x7f05481d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.265 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 >> v1:172.21.15.186:6789/0 conn(0x7f05481d7cd0 legacy=0x7f05481d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 1 monclient: found mon.a 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0548145740 con 0x7f054806ab80 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f05457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f054efa3640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f054efa3640 10 monclient: _renew_subs 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f054efa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0548145eb0 con 0x7f054806ab80 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.263+0000 7f054efa3640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:59.266 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.264+0000 7f054efa3640 10 monclient: _renew_subs 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.264+0000 7f054efa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.264+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481430b0 con 0x7f054806ab80 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.264+0000 7f054efa3640 1 librados: init done 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.264+0000 7f054efa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.265+0000 7f054efa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.265+0000 7f054efa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:59.267 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.265+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05480627b0 con 0x7f054806ab80 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f053c004a90 con 0x7f054806ab80 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7f053c004da0 con 0x7f054806ab80 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f05457fa640 10 monclient: dump: 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.571 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:34:59.572 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.569+0000 7f0546ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:59.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.570+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7f053c01ad70 con 0x7f054806ab80 2024-03-25T18:34:59.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.571+0000 7f0547fff640 10 monclient: get_auth_request con 0x7f052c06df60 auth_method 0 2024-03-25T18:34:59.573 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.571+0000 7f05457fa640 10 client.4686.objecter ms_handle_connect 0x7f052c06df60 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7f053c054800 con 0x7f054806ab80 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 10 client.4686.objecter ms_dispatch 0x7f0548000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 3 client.4686.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 3 client.4686.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f053c050140 con 0x7f054806ab80 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:59.587 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.585+0000 7f05457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:59.678 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:01.434 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.683 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:01.439 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.722 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:01.478 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.738 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:01.472 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.741 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:01.475 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.757 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:01.513 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.760 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:01.494 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.770 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:01.526 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.791 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:01.547 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:59.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.805+0000 7f054efa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:59.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.805+0000 7f054efa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:59.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.805+0000 7f054efa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:34:59.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.805+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0548070060 con 0x7f054806ab80 2024-03-25T18:34:59.865 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.864+0000 7f05457fa640 1 -- 172.21.15.176:0/91315893 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7f053c052240 con 0x7f054806ab80 2024-03-25T18:34:59.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.864+0000 7f05457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:59.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.864+0000 7f05457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:59.866 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:34:59.866 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.866+0000 7f054efa3640 10 librados: watch_flush enter 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 10 librados: watch_flush exit 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 10 client.4686.objecter successfully canceled tick 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 >> v1:172.21.15.176:6800/2598213983 conn(0x7f052c06df60 legacy=0x7f052c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 10 monclient: shutdown 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 >> v1:172.21.15.176:6789/0 conn(0x7f054806ab80 legacy=0x7f0548144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 shutdown_connections 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 >> 172.21.15.176:0/91315893 conn(0x7f05480b4a80 msgr2=0x7f0548070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 shutdown_connections 2024-03-25T18:34:59.869 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 -- 172.21.15.176:0/91315893 wait complete. 2024-03-25T18:34:59.870 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:34:59.867+0000 7f054efa3640 1 librados: shutdown 2024-03-25T18:34:59.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:06.689 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:06.799 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.798+0000 7fd847015640 10 monclient: get_monmap_and_config 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.798+0000 7fd847015640 10 monclient: build_initial_monmap 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.798+0000 7fd847015640 10 monclient: monmap: 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:06.799813+0000 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:06.799813+0000 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:06.800 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.799+0000 7fd847015640 1 Processor -- start 2024-03-25T18:35:06.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.799+0000 7fd847015640 1 -- start start 2024-03-25T18:35:06.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.799+0000 7fd847015640 10 monclient: init 2024-03-25T18:35:06.801 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient(hunting): picked mon.noname-b con 0x7fd84006f080 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient(hunting): picked mon.noname-a con 0x7fd8400704e0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient(hunting): picked mon.noname-c con 0x7fd8400ae6e0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8400718d0 con 0x7fd8400704e0 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84006fa30 con 0x7fd8400ae6e0 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd840070090 con 0x7fd84006f080 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:06.802 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.800+0000 7fd847015640 10 monclient(hunting): authenticate will time out at 8785.112253s 2024-03-25T18:35:06.810 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.809+0000 7fd8455b2640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd8400ae6e0 0x7fd84014bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:57600/0 (socket says 172.21.15.176:57600) 2024-03-25T18:35:06.810 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.809+0000 7fd8455b2640 1 -- 172.21.15.176:0/2380866082 learned_addr learned my addr 172.21.15.176:0/2380866082 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746422764 0 0) 0x7fd84006fa30 con 0x7fd8400ae6e0 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:35:06.828 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.827+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd830003520 con 0x7fd8400ae6e0 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851236547 0 0) 0x7fd8400718d0 con 0x7fd8400704e0 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:35:06.839 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.838+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84006fa30 con 0x7fd8400704e0 2024-03-25T18:35:06.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299749662 0 0) 0x7fd840070090 con 0x7fd84006f080 2024-03-25T18:35:06.841 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:35:06.842 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.840+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8400718d0 con 0x7fd84006f080 2024-03-25T18:35:06.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392693162 0 0) 0x7fd830003520 con 0x7fd8400ae6e0 2024-03-25T18:35:06.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 >> v1:172.21.15.176:6790/0 conn(0x7fd84006f080 legacy=0x7fd84013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:06.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 >> v1:172.21.15.176:6789/0 conn(0x7fd8400704e0 legacy=0x7fd840071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:06.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:06.850 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84006f510 con 0x7fd8400ae6e0 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd847015640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd838001a10 con 0x7fd8400ae6e0 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd83f7fe640 10 monclient: dump: 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.851 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.852 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.849+0000 7fd847015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd838001b70 con 0x7fd8400ae6e0 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 1 -- 172.21.15.176:0/2380866082 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd838001f80 con 0x7fd8400ae6e0 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd83f7fe640 10 monclient: dump: 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:06.856 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd847015640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:06.857 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.855+0000 7fd847015640 4 set_mon_vals no callback set 2024-03-25T18:35:06.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 10 monclient: shutdown 2024-03-25T18:35:06.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:06.858 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 1 -- 172.21.15.176:0/2380866082 >> v1:172.21.15.186:6789/0 conn(0x7fd8400ae6e0 legacy=0x7fd84014bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 1 -- 172.21.15.176:0/2380866082 shutdown_connections 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 1 -- 172.21.15.176:0/2380866082 >> 172.21.15.176:0/2380866082 conn(0x7fd84006b630 msgr2=0x7fd8400b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 1 -- 172.21.15.176:0/2380866082 shutdown_connections 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.856+0000 7fd847015640 1 -- 172.21.15.176:0/2380866082 wait complete. 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 10 monclient: build_initial_monmap 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 10 monclient: monmap: 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:06.859 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 librados: starting msgr at 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 librados: starting objecter 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 Processor -- start 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 -- start start 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 librados: setting wanted keys 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 1 librados: calling monclient init 2024-03-25T18:35:06.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.857+0000 7fd847015640 10 monclient: init 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 10 monclient(hunting): picked mon.noname-a con 0x7fd8400704e0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 10 monclient(hunting): picked mon.noname-b con 0x7fd8400ae6e0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 10 monclient(hunting): picked mon.noname-c con 0x7fd840133070 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:06.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8401410b0 con 0x7fd8400704e0 2024-03-25T18:35:06.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.858+0000 7fd847015640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8401417b0 con 0x7fd8400ae6e0 2024-03-25T18:35:06.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.859+0000 7fd847015640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84013e970 con 0x7fd840133070 2024-03-25T18:35:06.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.859+0000 7fd847015640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:06.862 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.859+0000 7fd847015640 10 monclient(hunting): authenticate will time out at 8785.170859s 2024-03-25T18:35:06.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.875+0000 7fd8455b2640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fd840133070 0x7fd8401dfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:59928/0 (socket says 172.21.15.176:59928) 2024-03-25T18:35:06.876 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.875+0000 7fd8455b2640 1 -- 172.21.15.176:0/889835385 learned_addr learned my addr 172.21.15.176:0/889835385 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:06.877 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.876+0000 7fd83d7fa640 10 client.?.objecter ms_handle_connect 0x7fd840133070 2024-03-25T18:35:06.877 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.876+0000 7fd83d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:06.879 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.879+0000 7fd83d7fa640 10 client.?.objecter ms_handle_connect 0x7fd8400704e0 2024-03-25T18:35:06.880 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.879+0000 7fd83d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:06.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170626514 0 0) 0x7fd8401410b0 con 0x7fd8400704e0 2024-03-25T18:35:06.890 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:35:06.891 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.889+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd824004eb0 con 0x7fd8400704e0 2024-03-25T18:35:06.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.892+0000 7fd83d7fa640 10 client.?.objecter ms_handle_connect 0x7fd8400ae6e0 2024-03-25T18:35:06.892 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.892+0000 7fd83d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020435408 0 0) 0x7fd84013e970 con 0x7fd840133070 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:35:06.898 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.897+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8401410b0 con 0x7fd840133070 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733448242 0 0) 0x7fd8401410b0 con 0x7fd840133070 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd84013e970 con 0x7fd840133070 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd838001e80 con 0x7fd840133070 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 10 monclient(hunting): dump: 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:06.907 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:06.908 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:06.908 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.908 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.905+0000 7fd83d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180754252 0 0) 0x7fd8401417b0 con 0x7fd8400ae6e0 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:35:06.910 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.909+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8401410b0 con 0x7fd8400ae6e0 2024-03-25T18:35:06.912 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1096491103 0 0) 0x7fd824004eb0 con 0x7fd8400704e0 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8401417b0 con 0x7fd8400704e0 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd8340018f0 con 0x7fd8400704e0 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.911+0000 7fd83d7fa640 10 monclient(hunting): dump: 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:06.913 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:06.914 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:06.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1973116133 0 0) 0x7fd8401417b0 con 0x7fd8400704e0 2024-03-25T18:35:06.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 >> v1:172.21.15.176:6790/0 conn(0x7fd840133070 legacy=0x7fd8401dfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:06.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 >> v1:172.21.15.186:6789/0 conn(0x7fd8400ae6e0 legacy=0x7fd8401d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:06.920 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 1 monclient: found mon.a 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84013eb00 con 0x7fd8400704e0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd83d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 10 monclient: _renew_subs 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84013ecf0 con 0x7fd8400704e0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 10 monclient: _renew_subs 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8401e18f0 con 0x7fd8400704e0 2024-03-25T18:35:06.921 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.918+0000 7fd847015640 1 librados: init done 2024-03-25T18:35:06.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.919+0000 7fd847015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:06.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.919+0000 7fd847015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:06.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.919+0000 7fd847015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:06.922 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:06.919+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd840141940 con 0x7fd8400704e0 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd834003610 con 0x7fd8400704e0 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd834003920 con 0x7fd8400704e0 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83d7fa640 10 monclient: dump: 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:07.236 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.234+0000 7fd83effd640 4 set_mon_vals no callback set 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.235+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fd834019930 con 0x7fd8400704e0 2024-03-25T18:35:07.237 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.236+0000 7fd83ffff640 10 monclient: get_auth_request con 0x7fd82406df50 auth_method 0 2024-03-25T18:35:07.238 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.236+0000 7fd83d7fa640 10 client.4692.objecter ms_handle_connect 0x7fd82406df50 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fd8340535f0 con 0x7fd8400704e0 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 10 client.4692.objecter ms_dispatch 0x7fd840000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd83404ef30 con 0x7fd8400704e0 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:07.244 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.243+0000 7fd83d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:07.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.478+0000 7fd847015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:07.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.478+0000 7fd847015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:07.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.478+0000 7fd847015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:07.479 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.478+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd840074d80 con 0x7fd8400704e0 2024-03-25T18:35:07.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.516+0000 7fd83d7fa640 1 -- 172.21.15.176:0/889835385 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fd840074d80 con 0x7fd8400704e0 2024-03-25T18:35:07.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.516+0000 7fd83d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:07.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.516+0000 7fd83d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:07.518 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:07.518 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 10 librados: watch_flush enter 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 10 librados: watch_flush exit 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 10 client.4692.objecter successfully canceled tick 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 >> v1:172.21.15.176:6800/2598213983 conn(0x7fd82406df50 legacy=0x7fd824070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:07.520 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 10 monclient: shutdown 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 >> v1:172.21.15.176:6789/0 conn(0x7fd8400704e0 legacy=0x7fd840142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 shutdown_connections 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 >> 172.21.15.176:0/889835385 conn(0x7fd84006b630 msgr2=0x7fd84013c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 shutdown_connections 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 -- 172.21.15.176:0/889835385 wait complete. 2024-03-25T18:35:07.521 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:07.519+0000 7fd847015640 1 librados: shutdown 2024-03-25T18:35:07.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:14.340 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:14.460 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.459+0000 7fc75f544640 10 monclient: get_monmap_and_config 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.459+0000 7fc75f544640 10 monclient: build_initial_monmap 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.459+0000 7fc75f544640 10 monclient: monmap: 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:14.460298+0000 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:14.460298+0000 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:14.461 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:14.462 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:14.462 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.460+0000 7fc75f544640 1 Processor -- start 2024-03-25T18:35:14.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.460+0000 7fc75f544640 1 -- start start 2024-03-25T18:35:14.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.460+0000 7fc75f544640 10 monclient: init 2024-03-25T18:35:14.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.460+0000 7fc75f544640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:14.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.460+0000 7fc75f544640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:14.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-c con 0x7fc7580a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:14.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-a con 0x7fc75813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-b con 0x7fc7581430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc758142520 con 0x7fc75813f710 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7580b28a0 con 0x7fc7580a98f0 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7580b2a30 con 0x7fc7581430c0 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:14.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.461+0000 7fc75f544640 10 monclient(hunting): authenticate will time out at 8792.773041s 2024-03-25T18:35:14.471 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.470+0000 7fc75dae1640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc7581430c0 0x7fc75814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39666/0 (socket says 172.21.15.176:39666) 2024-03-25T18:35:14.472 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.470+0000 7fc75dae1640 1 -- 172.21.15.176:0/2876014880 learned_addr learned my addr 172.21.15.176:0/2876014880 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.487+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517517689 0 0) 0x7fc758142520 con 0x7fc75813f710 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:35:14.489 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744003520 con 0x7fc75813f710 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552486299 0 0) 0x7fc7580b2a30 con 0x7fc7581430c0 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:35:14.490 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.488+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc758142520 con 0x7fc7581430c0 2024-03-25T18:35:14.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946715920 0 0) 0x7fc7580b28a0 con 0x7fc7580a98f0 2024-03-25T18:35:14.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.492 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:35:14.493 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.491+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7580b2a30 con 0x7fc7580a98f0 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.497+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691500863 0 0) 0x7fc758142520 con 0x7fc7581430c0 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.497+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 >> v1:172.21.15.186:6789/0 conn(0x7fc7580a98f0 legacy=0x7fc7580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.497+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 >> v1:172.21.15.176:6789/0 conn(0x7fc75813f710 legacy=0x7fc7581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 1 monclient: found mon.noname-b 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:14.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7580b44c0 con 0x7fc7581430c0 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: _finish_auth 0 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc7500019f0 con 0x7fc7581430c0 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc74ffff640 10 monclient: dump: 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.500 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc75f544640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:35:14.501 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.498+0000 7fc75f544640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:14.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc750001b50 con 0x7fc7581430c0 2024-03-25T18:35:14.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:14.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 1 -- 172.21.15.176:0/2876014880 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc750001f60 con 0x7fc7581430c0 2024-03-25T18:35:14.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc75f544640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:14.512 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc75f544640 4 set_mon_vals no callback set 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.511+0000 7fc74ffff640 10 monclient: dump: 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:14.513 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:14.514 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:14.514 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 10 monclient: shutdown 2024-03-25T18:35:14.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:14.516 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 1 -- 172.21.15.176:0/2876014880 >> v1:172.21.15.176:6790/0 conn(0x7fc7581430c0 legacy=0x7fc75814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 1 -- 172.21.15.176:0/2876014880 shutdown_connections 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 1 -- 172.21.15.176:0/2876014880 >> 172.21.15.176:0/2876014880 conn(0x7fc7580b7ea0 msgr2=0x7fc7580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 1 -- 172.21.15.176:0/2876014880 shutdown_connections 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.512+0000 7fc75f544640 1 -- 172.21.15.176:0/2876014880 wait complete. 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 10 monclient: build_initial_monmap 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 10 monclient: monmap: 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:14.517 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 librados: starting msgr at 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 librados: starting objecter 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 Processor -- start 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 -- start start 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 librados: setting wanted keys 2024-03-25T18:35:14.518 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 1 librados: calling monclient init 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.513+0000 7fc75f544640 10 monclient: init 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-c con 0x7fc7580a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-a con 0x7fc75813f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient(hunting): picked mon.noname-b con 0x7fc7581430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7581d73f0 con 0x7fc75813f710 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7581395e0 con 0x7fc7581430c0 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7581398c0 con 0x7fc7580a98f0 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:14.519 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.514+0000 7fc75f544640 10 monclient(hunting): authenticate will time out at 8792.826328s 2024-03-25T18:35:14.525 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.524+0000 7fc75d2e0640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fc7580a98f0 0x7fc7581d7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39710/0 (socket says 172.21.15.176:39710) 2024-03-25T18:35:14.526 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.524+0000 7fc75d2e0640 1 -- 172.21.15.176:0/4236654438 learned_addr learned my addr 172.21.15.176:0/4236654438 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:14.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.528+0000 7fc74dffb640 10 client.?.objecter ms_handle_connect 0x7fc7580a98f0 2024-03-25T18:35:14.529 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.528+0000 7fc74dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:14.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.533+0000 7fc74dffb640 10 client.?.objecter ms_handle_connect 0x7fc75813f710 2024-03-25T18:35:14.534 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.533+0000 7fc74dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:14.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.536+0000 7fc74dffb640 10 client.?.objecter ms_handle_connect 0x7fc7581430c0 2024-03-25T18:35:14.537 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.536+0000 7fc74dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875385283 0 0) 0x7fc7581398c0 con 0x7fc7580a98f0 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:35:14.539 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.537+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc73c004eb0 con 0x7fc7580a98f0 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969685604 0 0) 0x7fc7581d73f0 con 0x7fc75813f710 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:35:14.543 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.542+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7581398c0 con 0x7fc75813f710 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51913550 0 0) 0x7fc7581395e0 con 0x7fc7581430c0 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:35:14.552 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.550+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7581d73f0 con 0x7fc7581430c0 2024-03-25T18:35:14.554 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1134001055 0 0) 0x7fc7581398c0 con 0x7fc75813f710 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7581395e0 con 0x7fc75813f710 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2628185113 0 0) 0x7fc73c004eb0 con 0x7fc7580a98f0 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7581398c0 con 0x7fc7580a98f0 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc754001a10 con 0x7fc7580a98f0 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 10 monclient(hunting): dump: 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.555 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.556 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.553+0000 7fc74dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:14.562 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381262708 0 0) 0x7fc7581398c0 con 0x7fc7580a98f0 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 >> v1:172.21.15.186:6789/0 conn(0x7fc7581430c0 legacy=0x7fc758137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 >> v1:172.21.15.176:6789/0 conn(0x7fc75813f710 legacy=0x7fc7581d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 1 monclient: found mon.c 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758137b40 con 0x7fc7580a98f0 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:14.563 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc74dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc75f544640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc75f544640 10 monclient: _renew_subs 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc75f544640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.560+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7581d7f90 con 0x7fc7580a98f0 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.561+0000 7fc75f544640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.561+0000 7fc75f544640 10 monclient: _renew_subs 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.561+0000 7fc75f544640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.561+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7581d4b60 con 0x7fc7580a98f0 2024-03-25T18:35:14.564 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.561+0000 7fc75f544640 1 librados: init done 2024-03-25T18:35:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.562+0000 7fc75f544640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.562+0000 7fc75f544640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.562+0000 7fc75f544640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:14.565 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.562+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7580627b0 con 0x7fc7580a98f0 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc754001c70 con 0x7fc7580a98f0 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fc754003a60 con 0x7fc7580a98f0 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:14.859 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74dffb640 10 monclient: dump: 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:14.860 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.857+0000 7fc74f7fe640 4 set_mon_vals no callback set 2024-03-25T18:35:14.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.858+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fc754003c80 con 0x7fc7580a98f0 2024-03-25T18:35:14.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.859+0000 7fc75cadf640 10 monclient: get_auth_request con 0x7fc73c06e0c0 auth_method 0 2024-03-25T18:35:14.861 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.859+0000 7fc74dffb640 10 client.4733.objecter ms_handle_connect 0x7fc73c06e0c0 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fc7540548b0 con 0x7fc7580a98f0 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 10 client.4733.objecter ms_dispatch 0x7fc758000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7540503c0 con 0x7fc7580a98f0 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:14.866 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:14.864+0000 7fc74dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:15.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.108+0000 7fc75f544640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:15.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.108+0000 7fc75f544640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:15.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.108+0000 7fc75f544640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:15.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.108+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc758142290 con 0x7fc7580a98f0 2024-03-25T18:35:15.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.142+0000 7fc74dffb640 1 -- 172.21.15.176:0/4236654438 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fc7540522f0 con 0x7fc7580a98f0 2024-03-25T18:35:15.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.142+0000 7fc74dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:15.144 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.142+0000 7fc74dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:15.144 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:15.144 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 10 librados: watch_flush enter 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 10 librados: watch_flush exit 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 10 client.4733.objecter successfully canceled tick 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 >> v1:172.21.15.176:6800/2598213983 conn(0x7fc73c06e0c0 legacy=0x7fc73c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:15.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 10 monclient: shutdown 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 >> v1:172.21.15.176:6790/0 conn(0x7fc7580a98f0 legacy=0x7fc7581d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 shutdown_connections 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 >> 172.21.15.176:0/4236654438 conn(0x7fc7580b7ea0 msgr2=0x7fc758074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 shutdown_connections 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 -- 172.21.15.176:0/4236654438 wait complete. 2024-03-25T18:35:15.148 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:15.145+0000 7fc75f544640 1 librados: shutdown 2024-03-25T18:35:15.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:19.715 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:21.471 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.725 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:21.482 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.757 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:21.513 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.779 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:21.536 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.782 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:21.516 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.786 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:21.521 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.793 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:21.549 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.816 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:21.572 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:19.817 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:21.552 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:21.965 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:22.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.080+0000 7fa88a3a8640 10 monclient: get_monmap_and_config 2024-03-25T18:35:22.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.080+0000 7fa88a3a8640 10 monclient: build_initial_monmap 2024-03-25T18:35:22.081 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.080+0000 7fa88a3a8640 10 monclient: monmap: 2024-03-25T18:35:22.081 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:22.081 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:22.081694+0000 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:22.081694+0000 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:22.082 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.081+0000 7fa88a3a8640 1 Processor -- start 2024-03-25T18:35:22.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.081+0000 7fa88a3a8640 1 -- start start 2024-03-25T18:35:22.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.081+0000 7fa88a3a8640 10 monclient: init 2024-03-25T18:35:22.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:22.083 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8840a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa88413f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8841430c0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa884142520 con 0x7fa8840a98f0 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8840b28a0 con 0x7fa88413f710 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8840b2a30 con 0x7fa8841430c0 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:22.084 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.082+0000 7fa88a3a8640 10 monclient(hunting): authenticate will time out at 8800.394499s 2024-03-25T18:35:22.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.094+0000 7fa883fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa8840a98f0 0x7fa8840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:47712/0 (socket says 172.21.15.176:47712) 2024-03-25T18:35:22.095 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.094+0000 7fa883fff640 1 -- 172.21.15.176:0/49371201 learned_addr learned my addr 172.21.15.176:0/49371201 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:22.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248983333 0 0) 0x7fa884142520 con 0x7fa8840a98f0 2024-03-25T18:35:22.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:22.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:22.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:22.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:35:22.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.107+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa86c003520 con 0x7fa8840a98f0 2024-03-25T18:35:22.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54621626 0 0) 0x7fa8840b2a30 con 0x7fa8841430c0 2024-03-25T18:35:22.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:22.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:22.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:22.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:35:22.116 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.114+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884142520 con 0x7fa8841430c0 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.116+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 407792100 0 0) 0x7fa86c003520 con 0x7fa8840a98f0 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 >> v1:172.21.15.176:6790/0 conn(0x7fa8841430c0 legacy=0x7fa88414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 >> v1:172.21.15.186:6789/0 conn(0x7fa88413f710 legacy=0x7fa8841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 1 monclient: found mon.noname-a 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:35:22.118 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8840b44c0 con 0x7fa8840a98f0 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa87c001a10 con 0x7fa8840a98f0 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa88a3a8640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa88a3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa882ffd640 10 monclient: dump: 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.119 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:22.120 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:22.120 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:22.120 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.120 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.117+0000 7fa88a3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa87c001b70 con 0x7fa8840a98f0 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 1 -- 172.21.15.176:0/49371201 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa87c001f80 con 0x7fa8840a98f0 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa882ffd640 10 monclient: dump: 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.122 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa88a3a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:22.123 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa88a3a8640 4 set_mon_vals no callback set 2024-03-25T18:35:22.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa88a3a8640 10 monclient: shutdown 2024-03-25T18:35:22.125 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa88a3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.121+0000 7fa88a3a8640 1 -- 172.21.15.176:0/49371201 >> v1:172.21.15.176:6789/0 conn(0x7fa8840a98f0 legacy=0x7fa8840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 1 -- 172.21.15.176:0/49371201 shutdown_connections 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 1 -- 172.21.15.176:0/49371201 >> 172.21.15.176:0/49371201 conn(0x7fa8840b7ea0 msgr2=0x7fa8840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 1 -- 172.21.15.176:0/49371201 shutdown_connections 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 1 -- 172.21.15.176:0/49371201 wait complete. 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 10 monclient: build_initial_monmap 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.122+0000 7fa88a3a8640 10 monclient: monmap: 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:22.126 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 librados: starting msgr at 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 librados: starting objecter 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 Processor -- start 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 -- start start 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 librados: setting wanted keys 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 1 librados: calling monclient init 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.123+0000 7fa88a3a8640 10 monclient: init 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8840a98f0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa88413f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8841430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8841db2f0 con 0x7fa8840a98f0 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa884139180 con 0x7fa8841430c0 2024-03-25T18:35:22.127 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa884137670 con 0x7fa88413f710 2024-03-25T18:35:22.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:22.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.124+0000 7fa88a3a8640 10 monclient(hunting): authenticate will time out at 8800.436286s 2024-03-25T18:35:22.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.126+0000 7fa8837fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa88413f710 0x7fa884132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:44354/0 (socket says 172.21.15.176:44354) 2024-03-25T18:35:22.128 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.126+0000 7fa8837fe640 1 -- 172.21.15.176:0/941269685 learned_addr learned my addr 172.21.15.176:0/941269685 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:22.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.130+0000 7fa880ff9640 10 client.?.objecter ms_handle_connect 0x7fa88413f710 2024-03-25T18:35:22.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.130+0000 7fa880ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:22.137 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.137+0000 7fa880ff9640 10 client.?.objecter ms_handle_connect 0x7fa8840a98f0 2024-03-25T18:35:22.138 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.137+0000 7fa880ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641193378 0 0) 0x7fa884137670 con 0x7fa88413f710 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:35:22.142 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.141+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa870004eb0 con 0x7fa88413f710 2024-03-25T18:35:22.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.142+0000 7fa880ff9640 10 client.?.objecter ms_handle_connect 0x7fa8841430c0 2024-03-25T18:35:22.143 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.142+0000 7fa880ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779348490 0 0) 0x7fa884139180 con 0x7fa8841430c0 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:35:22.154 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884137670 con 0x7fa8841430c0 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525766802 0 0) 0x7fa870004eb0 con 0x7fa88413f710 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa884139180 con 0x7fa88413f710 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa878001a30 con 0x7fa88413f710 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): dump: 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.155 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.153+0000 7fa880ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150550472 0 0) 0x7fa8841db2f0 con 0x7fa8840a98f0 2024-03-25T18:35:22.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:22.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:22.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:22.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:35:22.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.155+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa870004eb0 con 0x7fa8840a98f0 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 582236836 0 0) 0x7fa870004eb0 con 0x7fa8840a98f0 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8841db2f0 con 0x7fa8840a98f0 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa87c001e80 con 0x7fa8840a98f0 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.172+0000 7fa880ff9640 10 monclient(hunting): dump: 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.173 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:22.174 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.174 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:22.174 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:22.174 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:22.174 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3903952601 0 0) 0x7fa884139180 con 0x7fa88413f710 2024-03-25T18:35:22.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 >> v1:172.21.15.186:6789/0 conn(0x7fa8841430c0 legacy=0x7fa8841e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.175 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 >> v1:172.21.15.176:6789/0 conn(0x7fa8840a98f0 legacy=0x7fa8841dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 1 monclient: found mon.c 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884137200 con 0x7fa88413f710 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 5 monclient: authenticate success, global_id 4739 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 10 monclient: _renew_subs 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8840b7290 con 0x7fa88413f710 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 10 client.4739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 10 monclient: _renew_subs 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:22.176 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa884133680 con 0x7fa88413f710 2024-03-25T18:35:22.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.174+0000 7fa88a3a8640 1 librados: init done 2024-03-25T18:35:22.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.175+0000 7fa88a3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:22.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.175+0000 7fa88a3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:22.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.175+0000 7fa88a3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:22.177 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.175+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8840627b0 con 0x7fa88413f710 2024-03-25T18:35:22.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa878001c70 con 0x7fa88413f710 2024-03-25T18:35:22.468 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa878003a60 con 0x7fa88413f710 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa880ff9640 10 monclient: dump: 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:22.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.467+0000 7fa8827fc640 4 set_mon_vals no callback set 2024-03-25T18:35:22.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.468+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa878003c40 con 0x7fa88413f710 2024-03-25T18:35:22.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.469+0000 7fa883fff640 10 monclient: get_auth_request con 0x7fa87006dfd0 auth_method 0 2024-03-25T18:35:22.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.469+0000 7fa880ff9640 10 client.4739.objecter ms_handle_connect 0x7fa87006dfd0 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fa8780548b0 con 0x7fa88413f710 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 10 client.4739.objecter ms_dispatch 0x7fa884000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 3 client.4739.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 3 client.4739.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 20 client.4739.objecter dump_active .. 0 homeless 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8780503c0 con 0x7fa88413f710 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:22.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.474+0000 7fa880ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:22.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.712+0000 7fa88a3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:22.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.712+0000 7fa88a3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:22.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.712+0000 7fa88a3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:22.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.712+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa88413e2b0 con 0x7fa88413f710 2024-03-25T18:35:22.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.749+0000 7fa880ff9640 1 -- 172.21.15.176:0/941269685 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fa8780522f0 con 0x7fa88413f710 2024-03-25T18:35:22.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.749+0000 7fa880ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:22.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.749+0000 7fa880ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:22.750 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:22.750 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 10 librados: watch_flush enter 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 10 librados: watch_flush exit 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 20 client.4739.objecter shutdown clearing up homeless session... 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 10 client.4739.objecter successfully canceled tick 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa87006dfd0 legacy=0x7fa870070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.753 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 10 monclient: shutdown 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.751+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 >> v1:172.21.15.176:6790/0 conn(0x7fa88413f710 legacy=0x7fa884132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.752+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 shutdown_connections 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.752+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 >> 172.21.15.176:0/941269685 conn(0x7fa8840b7ea0 msgr2=0x7fa884074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.752+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 shutdown_connections 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.752+0000 7fa88a3a8640 1 -- 172.21.15.176:0/941269685 wait complete. 2024-03-25T18:35:22.754 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:22.752+0000 7fa88a3a8640 1 librados: shutdown 2024-03-25T18:35:22.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:26.932 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:28.688 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.937 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:28.694 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.944 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:28.678 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.945 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:28.701 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.948 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:28.682 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.948 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:28.704 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.961 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:28.717 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.966 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:28.700 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:26.967 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:28.724 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:29.573 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:29.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.687+0000 7fa5b6123640 10 monclient: get_monmap_and_config 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.687+0000 7fa5b6123640 10 monclient: build_initial_monmap 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.687+0000 7fa5b6123640 10 monclient: monmap: 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:29.688965+0000 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:29.688965+0000 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:29.689 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:29.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.687+0000 7fa5b6123640 1 Processor -- start 2024-03-25T18:35:29.690 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 1 -- start start 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient: init 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b00b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b00b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00af940 con 0x7fa5b00b1730 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00b0a70 con 0x7fa5b014be90 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00b0d50 con 0x7fa5b00b4d60 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:29.691 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.688+0000 7fa5b6123640 10 monclient(hunting): authenticate will time out at 8808.001559s 2024-03-25T18:35:29.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.698+0000 7fa5af7fe640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa5b00b4d60 0x7fa5b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39072/0 (socket says 172.21.15.176:39072) 2024-03-25T18:35:29.700 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.698+0000 7fa5af7fe640 1 -- 172.21.15.176:0/887359562 learned_addr learned my addr 172.21.15.176:0/887359562 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:29.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.709+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633702299 0 0) 0x7fa5b00af940 con 0x7fa5b00b1730 2024-03-25T18:35:29.711 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.710+0000 7fa5ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:29.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.710+0000 7fa5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:29.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.710+0000 7fa5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:29.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.710+0000 7fa5ae7fc640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:35:29.712 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.710+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003520 con 0x7fa5b00b1730 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288956814 0 0) 0x7fa5b00b0d50 con 0x7fa5b00b4d60 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:35:29.713 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.711+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003840 con 0x7fa5b00b4d60 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3548510428 0 0) 0x7fa598003520 con 0x7fa5b00b1730 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 >> v1:172.21.15.176:6790/0 conn(0x7fa5b00b4d60 legacy=0x7fa5b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 >> v1:172.21.15.186:6789/0 conn(0x7fa5b014be90 legacy=0x7fa5b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 monclient: found mon.noname-a 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00b1030 con 0x7fa5b00b1730 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:29.727 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa5a4001670 con 0x7fa5b00b1730 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5b6123640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5b6123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5ae7fc640 10 monclient: dump: 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:29.728 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.724+0000 7fa5b6123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:29.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a40017d0 con 0x7fa5b00b1730 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 1 -- 172.21.15.176:0/887359562 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa5a4001be0 con 0x7fa5b00b1730 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5ae7fc640 10 monclient: dump: 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:29.747 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:29.748 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:29.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:29.748 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 4 set_mon_vals no callback set 2024-03-25T18:35:29.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 10 monclient: shutdown 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 1 -- 172.21.15.176:0/887359562 >> v1:172.21.15.176:6789/0 conn(0x7fa5b00b1730 legacy=0x7fa5b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 1 -- 172.21.15.176:0/887359562 shutdown_connections 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.745+0000 7fa5b6123640 1 -- 172.21.15.176:0/887359562 >> 172.21.15.176:0/887359562 conn(0x7fa5b00617a0 msgr2=0x7fa5b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 1 -- 172.21.15.176:0/887359562 shutdown_connections 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 1 -- 172.21.15.176:0/887359562 wait complete. 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 10 monclient: build_initial_monmap 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 10 monclient: monmap: 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:29.750 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 1 librados: starting msgr at 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.746+0000 7fa5b6123640 1 librados: starting objecter 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 Processor -- start 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 -- start start 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 librados: setting wanted keys 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 librados: calling monclient init 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient: init 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b00b1730 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b00b4d60 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:29.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b014be90 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:29.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b01406f0 con 0x7fa5b00b1730 2024-03-25T18:35:29.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b0140380 con 0x7fa5b014be90 2024-03-25T18:35:29.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b01372a0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:29.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.747+0000 7fa5b6123640 10 monclient(hunting): authenticate will time out at 8808.060817s 2024-03-25T18:35:29.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.758+0000 7fa5aeffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa5b00b4d60 0x7fa5b0135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:39120/0 (socket says 172.21.15.176:39120) 2024-03-25T18:35:29.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.758+0000 7fa5aeffd640 1 -- 172.21.15.176:0/2034538178 learned_addr learned my addr 172.21.15.176:0/2034538178 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:29.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.759+0000 7fa597fff640 10 client.?.objecter ms_handle_connect 0x7fa5b00b4d60 2024-03-25T18:35:29.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.759+0000 7fa597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:29.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.763+0000 7fa597fff640 10 client.?.objecter ms_handle_connect 0x7fa5b014be90 2024-03-25T18:35:29.765 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.763+0000 7fa597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:29.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.764+0000 7fa597fff640 10 client.?.objecter ms_handle_connect 0x7fa5b00b1730 2024-03-25T18:35:29.766 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.764+0000 7fa597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961467081 0 0) 0x7fa5b01372a0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:35:29.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.766+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c004eb0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544140190 0 0) 0x7fa59c004eb0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b01372a0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa5a4001c20 con 0x7fa5b00b4d60 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 10 monclient(hunting): dump: 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:29.779 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:29.780 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:29.780 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:29.780 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:29.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.777+0000 7fa597fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:29.781 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436630049 0 0) 0x7fa5b01406f0 con 0x7fa5b00b1730 2024-03-25T18:35:29.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:29.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:29.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:29.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:35:29.782 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.780+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c004eb0 con 0x7fa5b00b1730 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3005858033 0 0) 0x7fa5b01372a0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 >> v1:172.21.15.186:6789/0 conn(0x7fa5b014be90 legacy=0x7fa5b0136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 >> v1:172.21.15.176:6789/0 conn(0x7fa5b00b1730 legacy=0x7fa5b0134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 1 monclient: found mon.c 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b0137430 con 0x7fa5b00b4d60 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa5b6123640 5 monclient: authenticate success, global_id 4745 2024-03-25T18:35:29.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa5b6123640 10 monclient: _renew_subs 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.783+0000 7fa5b6123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b01355c0 con 0x7fa5b00b4d60 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 10 client.4745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 10 monclient: _renew_subs 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b0137e20 con 0x7fa5b00b4d60 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.784+0000 7fa5b6123640 1 librados: init done 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.785+0000 7fa5b6123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:29.787 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.785+0000 7fa5b6123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:29.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.785+0000 7fa5b6123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:29.788 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:29.785+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b01408a0 con 0x7fa5b00b4d60 2024-03-25T18:35:30.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a40038a0 con 0x7fa5b00b4d60 2024-03-25T18:35:30.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:30.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa5a4003ad0 con 0x7fa5b00b4d60 2024-03-25T18:35:30.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:30.107 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa597fff640 10 monclient: dump: 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:30.108 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.105+0000 7fa5adffb640 4 set_mon_vals no callback set 2024-03-25T18:35:30.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.106+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa5a4003cf0 con 0x7fa5b00b4d60 2024-03-25T18:35:30.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.106+0000 7fa5af7fe640 10 monclient: get_auth_request con 0x7fa59c06dd40 auth_method 0 2024-03-25T18:35:30.109 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.106+0000 7fa597fff640 10 client.4745.objecter ms_handle_connect 0x7fa59c06dd40 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fa5a4054830 con 0x7fa5b00b4d60 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 10 client.4745.objecter ms_dispatch 0x7fa5b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 3 client.4745.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 3 client.4745.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 20 client.4745.objecter dump_active .. 0 homeless 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5a4050340 con 0x7fa5b00b4d60 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:30.113 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.111+0000 7fa597fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:30.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.384+0000 7fa5b6123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:30.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.384+0000 7fa5b6123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:30.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.384+0000 7fa5b6123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.176:6790/0 2024-03-25T18:35:30.387 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.384+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 --> v1:172.21.15.176:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b0140510 con 0x7fa5b00b4d60 2024-03-25T18:35:30.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.413+0000 7fa597fff640 1 -- 172.21.15.176:0/2034538178 <== mon.2 v1:172.21.15.176:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fa5a4052270 con 0x7fa5b00b4d60 2024-03-25T18:35:30.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.413+0000 7fa597fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:30.415 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.413+0000 7fa597fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:30.416 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:30.416 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:30.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 10 librados: watch_flush enter 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 10 librados: watch_flush exit 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 20 client.4745.objecter shutdown clearing up homeless session... 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 10 client.4745.objecter successfully canceled tick 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa59c06dd40 legacy=0x7fa59c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 10 monclient: shutdown 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 >> v1:172.21.15.176:6790/0 conn(0x7fa5b00b4d60 legacy=0x7fa5b0135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 shutdown_connections 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 >> 172.21.15.176:0/2034538178 conn(0x7fa5b00617a0 msgr2=0x7fa5b013c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 shutdown_connections 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 -- 172.21.15.176:0/2034538178 wait complete. 2024-03-25T18:35:30.418 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:30.415+0000 7fa5b6123640 1 librados: shutdown 2024-03-25T18:35:30.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:37.236 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.349+0000 7efd85035640 10 monclient: get_monmap_and_config 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.349+0000 7efd85035640 10 monclient: build_initial_monmap 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.349+0000 7efd85035640 10 monclient: monmap: 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:37.350963+0000 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:37.350963+0000 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:37.351 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:37.352 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.350+0000 7efd85035640 1 Processor -- start 2024-03-25T18:35:37.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.350+0000 7efd85035640 1 -- start start 2024-03-25T18:35:37.352 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.350+0000 7efd85035640 10 monclient: init 2024-03-25T18:35:37.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:37.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:37.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient(hunting): picked mon.noname-c con 0x7efd800a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:37.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient(hunting): picked mon.noname-b con 0x7efd8013f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:37.353 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient(hunting): picked mon.noname-a con 0x7efd801430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:37.354 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd80142520 con 0x7efd801430c0 2024-03-25T18:35:37.354 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd800b28a0 con 0x7efd800a98f0 2024-03-25T18:35:37.354 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd800b2a30 con 0x7efd8013f710 2024-03-25T18:35:37.354 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:37.354 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.351+0000 7efd85035640 10 monclient(hunting): authenticate will time out at 8815.664393s 2024-03-25T18:35:37.360 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.358+0000 7efd77fff640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7efd8013f710 0x7efd801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:35804/0 (socket says 172.21.15.176:35804) 2024-03-25T18:35:37.360 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.358+0000 7efd77fff640 1 -- 172.21.15.176:0/2148443530 learned_addr learned my addr 172.21.15.176:0/2148443530 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:37.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213217871 0 0) 0x7efd800b2a30 con 0x7efd8013f710 2024-03-25T18:35:37.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:37.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:37.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:37.370 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:35:37.371 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.368+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd60003520 con 0x7efd8013f710 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589347283 0 0) 0x7efd800b28a0 con 0x7efd800a98f0 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:35:37.381 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.379+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800b2a30 con 0x7efd800a98f0 2024-03-25T18:35:37.389 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892854012 0 0) 0x7efd60003520 con 0x7efd8013f710 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 >> v1:172.21.15.186:6789/0 conn(0x7efd800a98f0 legacy=0x7efd800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 >> v1:172.21.15.176:6789/0 conn(0x7efd801430c0 legacy=0x7efd8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: _finish_hunting 0 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 monclient: found mon.noname-b 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.176:6790/0 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 --> v1:172.21.15.176:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd800b44c0 con 0x7efd8013f710 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: _finish_auth 0 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: _check_auth_tickets 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd68001690 con 0x7efd8013f710 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:37.390 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd7e59c640 10 monclient: dump: 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd85035640 5 monclient: authenticate success, global_id 4748 2024-03-25T18:35:37.391 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.387+0000 7efd85035640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:37.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.2 v1:172.21.15.176:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd680017f0 con 0x7efd8013f710 2024-03-25T18:35:37.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:37.400 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 1 -- 172.21.15.176:0/2148443530 <== mon.2 v1:172.21.15.176:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd68001c00 con 0x7efd8013f710 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd85035640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd85035640 4 set_mon_vals no callback set 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.398+0000 7efd7e59c640 10 monclient: dump: 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:37.401 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 10 monclient: shutdown 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 1 -- 172.21.15.176:0/2148443530 >> v1:172.21.15.176:6790/0 conn(0x7efd8013f710 legacy=0x7efd801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 1 -- 172.21.15.176:0/2148443530 shutdown_connections 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 1 -- 172.21.15.176:0/2148443530 >> 172.21.15.176:0/2148443530 conn(0x7efd800b7ea0 msgr2=0x7efd800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 1 -- 172.21.15.176:0/2148443530 shutdown_connections 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.399+0000 7efd85035640 1 -- 172.21.15.176:0/2148443530 wait complete. 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.400+0000 7efd85035640 10 monclient: build_initial_monmap 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.400+0000 7efd85035640 10 monclient: monmap: 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:37.404 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 librados: starting msgr at 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 librados: starting objecter 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 Processor -- start 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 -- start start 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 librados: setting wanted keys 2024-03-25T18:35:37.405 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 1 librados: calling monclient init 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient: init 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient(hunting): picked mon.noname-c con 0x7efd800a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient(hunting): picked mon.noname-a con 0x7efd8013f710 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.401+0000 7efd85035640 10 monclient(hunting): picked mon.noname-b con 0x7efd801430c0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.402+0000 7efd85035640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd80138e90 con 0x7efd8013f710 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.402+0000 7efd85035640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd801399c0 con 0x7efd801430c0 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.402+0000 7efd85035640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd800abed0 con 0x7efd800a98f0 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.402+0000 7efd85035640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:37.406 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.402+0000 7efd85035640 10 monclient(hunting): authenticate will time out at 8815.714936s 2024-03-25T18:35:37.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.407+0000 7efd7f59e640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7efd801430c0 0x7efd801e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:41492/0 (socket says 172.21.15.176:41492) 2024-03-25T18:35:37.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.407+0000 7efd7f59e640 1 -- 172.21.15.176:0/2512239339 learned_addr learned my addr 172.21.15.176:0/2512239339 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:37.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.408+0000 7efd777fe640 10 client.?.objecter ms_handle_connect 0x7efd801430c0 2024-03-25T18:35:37.409 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.408+0000 7efd777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:37.412 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.410+0000 7efd777fe640 10 client.?.objecter ms_handle_connect 0x7efd800a98f0 2024-03-25T18:35:37.412 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.410+0000 7efd777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208687916 0 0) 0x7efd801399c0 con 0x7efd801430c0 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:35:37.470 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.468+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64004eb0 con 0x7efd801430c0 2024-03-25T18:35:37.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814347577 0 0) 0x7efd64004eb0 con 0x7efd801430c0 2024-03-25T18:35:37.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd801399c0 con 0x7efd801430c0 2024-03-25T18:35:37.476 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd78001c30 con 0x7efd801430c0 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 10 monclient(hunting): dump: 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.477 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.474+0000 7efd777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061667437 0 0) 0x7efd800abed0 con 0x7efd800a98f0 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 monclient(hunting): my global_id is 4751 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64004eb0 con 0x7efd800a98f0 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 client.?.objecter ms_handle_connect 0x7efd8013f710 2024-03-25T18:35:37.484 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.482+0000 7efd777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95723222 0 0) 0x7efd80138e90 con 0x7efd8013f710 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:35:37.499 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.497+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800abed0 con 0x7efd8013f710 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2612230307 0 0) 0x7efd801399c0 con 0x7efd801430c0 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 >> v1:172.21.15.176:6790/0 conn(0x7efd800a98f0 legacy=0x7efd80138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 >> v1:172.21.15.176:6789/0 conn(0x7efd8013f710 legacy=0x7efd800ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 1 monclient: found mon.b 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:37.502 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd801390c0 con 0x7efd801430c0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 10 monclient: _renew_subs 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd800aca00 con 0x7efd801430c0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 10 monclient: _renew_subs 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd801365e0 con 0x7efd801430c0 2024-03-25T18:35:37.503 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.500+0000 7efd85035640 1 librados: init done 2024-03-25T18:35:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.501+0000 7efd85035640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.501+0000 7efd85035640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.501+0000 7efd85035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:37.504 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.501+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd800627b0 con 0x7efd801430c0 2024-03-25T18:35:37.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.721+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd78003970 con 0x7efd801430c0 2024-03-25T18:35:37.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.721+0000 7efd777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:37.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.721+0000 7efd7dd9b640 4 set_mon_vals no callback set 2024-03-25T18:35:37.778 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.777+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7efd78003d80 con 0x7efd801430c0 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.777+0000 7efd777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.777+0000 7efd777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.777+0000 7efd777fe640 10 monclient: dump: 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:37.779 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:37.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.778+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7efd78003f60 con 0x7efd801430c0 2024-03-25T18:35:37.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.778+0000 7efd7ed9d640 10 monclient: get_auth_request con 0x7efd6406de10 auth_method 0 2024-03-25T18:35:37.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.778+0000 7efd777fe640 10 client.4738.objecter ms_handle_connect 0x7efd6406de10 2024-03-25T18:35:37.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.781+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7efd78054b10 con 0x7efd801430c0 2024-03-25T18:35:37.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.781+0000 7efd777fe640 10 client.4738.objecter ms_dispatch 0x7efd80000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:37.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.781+0000 7efd777fe640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:37.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.781+0000 7efd777fe640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:37.783 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.781+0000 7efd777fe640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:35:37.785 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.784+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd78050620 con 0x7efd801430c0 2024-03-25T18:35:37.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.784+0000 7efd777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:37.786 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:37.784+0000 7efd777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:38.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.036+0000 7efd85035640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:38.038 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.036+0000 7efd85035640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:38.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.036+0000 7efd85035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:38.039 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.036+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd8013e210 con 0x7efd801430c0 2024-03-25T18:35:38.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.060+0000 7efd777fe640 1 -- 172.21.15.176:0/2512239339 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7efd78052550 con 0x7efd801430c0 2024-03-25T18:35:38.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.060+0000 7efd777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:38.062 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.060+0000 7efd777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:38.063 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:38.063 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:38.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 10 librados: watch_flush enter 2024-03-25T18:35:38.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 10 librados: watch_flush exit 2024-03-25T18:35:38.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:35:38.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:35:38.065 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 >> v1:172.21.15.176:6800/2598213983 conn(0x7efd6406de10 legacy=0x7efd640702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 10 monclient: shutdown 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 >> v1:172.21.15.186:6789/0 conn(0x7efd801430c0 legacy=0x7efd801e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 shutdown_connections 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 >> 172.21.15.176:0/2512239339 conn(0x7efd800b7ea0 msgr2=0x7efd80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 shutdown_connections 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 -- 172.21.15.176:0/2512239339 wait complete. 2024-03-25T18:35:38.066 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:38.063+0000 7efd85035640 1 librados: shutdown 2024-03-25T18:35:38.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:44.761 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:46.517 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.805 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:46.539 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.808 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:46.565 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.810 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:46.544 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.841 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:11:46.597 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.849 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:46.584 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.866 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:46.622 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.877 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:46.634 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.886 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:44.909 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:11:46.665 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:44.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.985+0000 7fd0921c4640 10 monclient: get_monmap_and_config 2024-03-25T18:35:44.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.985+0000 7fd0921c4640 10 monclient: build_initial_monmap 2024-03-25T18:35:44.987 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.985+0000 7fd0921c4640 10 monclient: monmap: 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:44.987806+0000 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:44.987806+0000 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:44.988 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:44.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.986+0000 7fd0921c4640 1 Processor -- start 2024-03-25T18:35:44.989 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.986+0000 7fd0921c4640 1 -- start start 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.986+0000 7fd0921c4640 10 monclient: init 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd08c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd08c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd08c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c142520 con 0x7fd08c1430c0 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c0b28a0 con 0x7fd08c13f710 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c0b2a30 con 0x7fd08c0a98f0 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:44.990 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.987+0000 7fd0921c4640 10 monclient(hunting): authenticate will time out at 8823.300305s 2024-03-25T18:35:44.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.993+0000 7fd08affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd08c13f710 0x7fd08c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:55250/0 (socket says 172.21.15.176:55250) 2024-03-25T18:35:44.995 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:44.993+0000 7fd08affd640 1 -- 172.21.15.176:0/2584387983 learned_addr learned my addr 172.21.15.176:0/2584387983 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:45.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489346129 0 0) 0x7fd08c0b28a0 con 0x7fd08c13f710 2024-03-25T18:35:45.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:45.058 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:45.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:45.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:35:45.059 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.056+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd074003520 con 0x7fd08c13f710 2024-03-25T18:35:45.086 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273693210 0 0) 0x7fd074003520 con 0x7fd08c13f710 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 >> v1:172.21.15.176:6790/0 conn(0x7fd08c0a98f0 legacy=0x7fd08c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 >> v1:172.21.15.176:6789/0 conn(0x7fd08c1430c0 legacy=0x7fd08c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c0b44c0 con 0x7fd08c13f710 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd080001690 con 0x7fd08c13f710 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:45.087 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd08a7fc640 10 monclient: dump: 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd0921c4640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:35:45.088 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.084+0000 7fd0921c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 <== mon.1 v1:172.21.15.186:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0800017f0 con 0x7fd08c13f710 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 1 -- 172.21.15.176:0/2584387983 <== mon.1 v1:172.21.15.186:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd080001c00 con 0x7fd08c13f710 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd0921c4640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd0921c4640 4 set_mon_vals no callback set 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.098+0000 7fd08a7fc640 10 monclient: dump: 2024-03-25T18:35:45.100 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:45.101 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:45.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 10 monclient: shutdown 2024-03-25T18:35:45.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:45.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 1 -- 172.21.15.176:0/2584387983 >> v1:172.21.15.186:6789/0 conn(0x7fd08c13f710 legacy=0x7fd08c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 1 -- 172.21.15.176:0/2584387983 shutdown_connections 2024-03-25T18:35:45.102 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 1 -- 172.21.15.176:0/2584387983 >> 172.21.15.176:0/2584387983 conn(0x7fd08c0b7ea0 msgr2=0x7fd08c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 1 -- 172.21.15.176:0/2584387983 shutdown_connections 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.099+0000 7fd0921c4640 1 -- 172.21.15.176:0/2584387983 wait complete. 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.100+0000 7fd0921c4640 10 monclient: build_initial_monmap 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.100+0000 7fd0921c4640 10 monclient: monmap: 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.100+0000 7fd0921c4640 1 librados: starting msgr at 2024-03-25T18:35:45.103 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.100+0000 7fd0921c4640 1 librados: starting objecter 2024-03-25T18:35:45.104 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.100+0000 7fd0921c4640 1 Processor -- start 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 -- start start 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 librados: setting wanted keys 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 librados: calling monclient init 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient: init 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd08c0a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd08c13f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd08c1430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c13be70 con 0x7fd08c1430c0 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c131660 con 0x7fd08c13f710 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.101+0000 7fd0921c4640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c1317f0 con 0x7fd08c0a98f0 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.102+0000 7fd0921c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:45.105 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.102+0000 7fd0921c4640 10 monclient(hunting): authenticate will time out at 8823.414861s 2024-03-25T18:35:45.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.113+0000 7fd08affd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd08c13f710 0x7fd08c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:55274/0 (socket says 172.21.15.176:55274) 2024-03-25T18:35:45.115 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.113+0000 7fd08affd640 1 -- 172.21.15.176:0/4096356694 learned_addr learned my addr 172.21.15.176:0/4096356694 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:45.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.115+0000 7fd073fff640 10 client.?.objecter ms_handle_connect 0x7fd08c13f710 2024-03-25T18:35:45.117 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.115+0000 7fd073fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:45.123 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.122+0000 7fd073fff640 10 client.?.objecter ms_handle_connect 0x7fd08c0a98f0 2024-03-25T18:35:45.123 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.122+0000 7fd073fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:45.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116696343 0 0) 0x7fd08c131660 con 0x7fd08c13f710 2024-03-25T18:35:45.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:45.130 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:45.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:45.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:35:45.131 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.129+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd078004eb0 con 0x7fd08c13f710 2024-03-25T18:35:45.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641475387 0 0) 0x7fd08c1317f0 con 0x7fd08c0a98f0 2024-03-25T18:35:45.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:45.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:45.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:45.146 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 10 monclient(hunting): my global_id is 4757 2024-03-25T18:35:45.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.144+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd08c131660 con 0x7fd08c0a98f0 2024-03-25T18:35:45.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.145+0000 7fd073fff640 10 client.?.objecter ms_handle_connect 0x7fd08c1430c0 2024-03-25T18:35:45.147 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.145+0000 7fd073fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733655798 0 0) 0x7fd078004eb0 con 0x7fd08c13f710 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd08c1317f0 con 0x7fd08c13f710 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd080001c00 con 0x7fd08c13f710 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 10 monclient(hunting): dump: 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:45.156 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:45.157 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:45.157 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:45.157 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:45.157 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:45.157 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.154+0000 7fd073fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:45.159 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530616613 0 0) 0x7fd08c13be70 con 0x7fd08c1430c0 2024-03-25T18:35:45.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:45.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:45.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:45.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:35:45.160 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.158+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd078004eb0 con 0x7fd08c1430c0 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.177+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1876877314 0 0) 0x7fd08c1317f0 con 0x7fd08c13f710 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 >> v1:172.21.15.176:6790/0 conn(0x7fd08c0a98f0 legacy=0x7fd08c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 >> v1:172.21.15.176:6789/0 conn(0x7fd08c1430c0 legacy=0x7fd08c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 1 monclient: found mon.b 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c131980 con 0x7fd08c13f710 2024-03-25T18:35:45.180 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd073fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 5 monclient: authenticate success, global_id 4744 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 10 monclient: _renew_subs 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c135c60 con 0x7fd08c13f710 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 10 monclient: _renew_subs 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c0b7770 con 0x7fd08c13f710 2024-03-25T18:35:45.181 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.178+0000 7fd0921c4640 1 librados: init done 2024-03-25T18:35:45.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.179+0000 7fd0921c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:45.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.179+0000 7fd0921c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:45.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.179+0000 7fd0921c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.182 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.179+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08c1371d0 con 0x7fd08c13f710 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd080004a90 con 0x7fd08c13f710 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fd080004e20 con 0x7fd08c13f710 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd073fff640 10 monclient: dump: 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:45.462 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:45.463 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.459+0000 7fd089ffb640 4 set_mon_vals no callback set 2024-03-25T18:35:45.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.460+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fd080005040 con 0x7fd08c13f710 2024-03-25T18:35:45.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.461+0000 7fd08b7fe640 10 monclient: get_auth_request con 0x7fd07806de90 auth_method 0 2024-03-25T18:35:45.464 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.461+0000 7fd073fff640 10 client.4744.objecter ms_handle_connect 0x7fd07806de90 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.466+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fd080054950 con 0x7fd08c13f710 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.466+0000 7fd073fff640 10 client.4744.objecter ms_dispatch 0x7fd08c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.466+0000 7fd073fff640 3 client.4744.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.466+0000 7fd073fff640 3 client.4744.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.466+0000 7fd073fff640 20 client.4744.objecter dump_active .. 0 homeless 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.467+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd080050460 con 0x7fd08c13f710 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.467+0000 7fd073fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:45.469 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.467+0000 7fd073fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:45.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.714+0000 7fd0921c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:45.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.714+0000 7fd0921c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:45.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.714+0000 7fd0921c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.186:6789/0 2024-03-25T18:35:45.716 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.714+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 --> v1:172.21.15.186:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd08c0627b0 con 0x7fd08c13f710 2024-03-25T18:35:45.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.745+0000 7fd073fff640 1 -- 172.21.15.176:0/4096356694 <== mon.1 v1:172.21.15.186:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fd080052390 con 0x7fd08c13f710 2024-03-25T18:35:45.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.745+0000 7fd073fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:45.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.745+0000 7fd073fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:45.747 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:45.747 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 10 librados: watch_flush enter 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 10 librados: watch_flush exit 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 20 client.4744.objecter shutdown clearing up homeless session... 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 10 client.4744.objecter successfully canceled tick 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 >> v1:172.21.15.176:6800/2598213983 conn(0x7fd07806de90 legacy=0x7fd078070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 10 monclient: shutdown 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 >> v1:172.21.15.186:6789/0 conn(0x7fd08c13f710 legacy=0x7fd08c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 shutdown_connections 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.747+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 >> 172.21.15.176:0/4096356694 conn(0x7fd08c0b7ea0 msgr2=0x7fd08c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.748+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 shutdown_connections 2024-03-25T18:35:45.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.748+0000 7fd0921c4640 1 -- 172.21.15.176:0/4096356694 wait complete. 2024-03-25T18:35:45.751 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:45.748+0000 7fd0921c4640 1 librados: shutdown 2024-03-25T18:35:45.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:52.567 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:52.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.683+0000 7fa92d263640 10 monclient: get_monmap_and_config 2024-03-25T18:35:52.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.683+0000 7fa92d263640 10 monclient: build_initial_monmap 2024-03-25T18:35:52.685 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.683+0000 7fa92d263640 10 monclient: monmap: 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:35:52.685664+0000 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:35:52.685664+0000 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-b 2024-03-25T18:35:52.686 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.684+0000 7fa92d263640 1 Processor -- start 2024-03-25T18:35:52.687 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.684+0000 7fa92d263640 1 -- start start 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.684+0000 7fa92d263640 10 monclient: init 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280a98f0 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-c con 0x7fa92813f710 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-a con 0x7fa9281430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928142520 con 0x7fa9281430c0 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280b28a0 con 0x7fa92813f710 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280b2a30 con 0x7fa9280a98f0 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:52.688 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.685+0000 7fa92d263640 10 monclient(hunting): authenticate will time out at 8830.998744s 2024-03-25T18:35:52.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.696+0000 7fa926ffd640 1 --1- >> v1:172.21.15.176:6790/0 conn(0x7fa9280a98f0 0x7fa9280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6790/0 says I am v1:172.21.15.176:56052/0 (socket says 172.21.15.176:56052) 2024-03-25T18:35:52.698 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.696+0000 7fa926ffd640 1 -- 172.21.15.176:0/1426346032 learned_addr learned my addr 172.21.15.176:0/1426346032 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250842937 0 0) 0x7fa9280b28a0 con 0x7fa92813f710 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:35:52.718 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.716+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa910003520 con 0x7fa92813f710 2024-03-25T18:35:52.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307239127 0 0) 0x7fa9280b2a30 con 0x7fa9280a98f0 2024-03-25T18:35:52.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.720 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 10 monclient(hunting): my global_id is 4760 2024-03-25T18:35:52.721 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.718+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9280b28a0 con 0x7fa9280a98f0 2024-03-25T18:35:52.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482064858 0 0) 0x7fa928142520 con 0x7fa9281430c0 2024-03-25T18:35:52.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.723 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:35:52.724 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.721+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9280b2a30 con 0x7fa9281430c0 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048392864 0 0) 0x7fa9280b2a30 con 0x7fa9281430c0 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 >> v1:172.21.15.176:6790/0 conn(0x7fa9280a98f0 legacy=0x7fa9280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 >> v1:172.21.15.186:6789/0 conn(0x7fa92813f710 legacy=0x7fa9281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 monclient: found mon.noname-a 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:52.732 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.176:6789/0 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280b44c0 con 0x7fa9281430c0 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa920001930 con 0x7fa9281430c0 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa925ffb640 10 monclient: dump: 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.733 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.734 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.734 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.734 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa92d263640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:35:52.734 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.730+0000 7fa92d263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:52.746 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.0 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa920001a90 con 0x7fa9281430c0 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 1 -- 172.21.15.176:0/1426346032 <== mon.0 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa920001ee0 con 0x7fa9281430c0 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa92d263640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa92d263640 4 set_mon_vals no callback set 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.744+0000 7fa925ffb640 10 monclient: dump: 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.747 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.748 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.748 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 10 monclient: shutdown 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 1 -- 172.21.15.176:0/1426346032 >> v1:172.21.15.176:6789/0 conn(0x7fa9281430c0 legacy=0x7fa92814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 1 -- 172.21.15.176:0/1426346032 shutdown_connections 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 1 -- 172.21.15.176:0/1426346032 >> 172.21.15.176:0/1426346032 conn(0x7fa9280b7ea0 msgr2=0x7fa9280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 1 -- 172.21.15.176:0/1426346032 shutdown_connections 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.745+0000 7fa92d263640 1 -- 172.21.15.176:0/1426346032 wait complete. 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.746+0000 7fa92d263640 10 monclient: build_initial_monmap 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.746+0000 7fa92d263640 10 monclient: monmap: 2024-03-25T18:35:52.749 INFO:teuthology.misc.health.smithi176.stderr:epoch 0 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:last_changed 0.000000 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:created 0.000000 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 1 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.noname-a 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.noname-b 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.noname-c 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 librados: starting msgr at 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 librados: starting objecter 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 Processor -- start 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 -- start start 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 librados: setting wanted keys 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 1 librados: calling monclient init 2024-03-25T18:35:52.750 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.747+0000 7fa92d263640 10 monclient: init 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280a98f0 addr v1:172.21.15.186:6789/0 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-c con 0x7fa92813f710 addr v1:172.21.15.176:6790/0 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient(hunting): picked mon.noname-a con 0x7fa9281430c0 addr v1:172.21.15.176:6789/0 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928138eb0 con 0x7fa9281430c0 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928139920 con 0x7fa9280a98f0 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 1 -- --> v1:172.21.15.176:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280abde0 con 0x7fa92813f710 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:52.752 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.748+0000 7fa92d263640 10 monclient(hunting): authenticate will time out at 8831.061285s 2024-03-25T18:35:52.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.757+0000 7fa926ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa9280a98f0 0x7fa928138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.176:60148/0 (socket says 172.21.15.176:60148) 2024-03-25T18:35:52.759 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.757+0000 7fa926ffd640 1 -- 172.21.15.176:0/1105769292 learned_addr learned my addr 172.21.15.176:0/1105769292 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-25T18:35:52.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.759+0000 7fa90f7fe640 10 client.?.objecter ms_handle_connect 0x7fa9280a98f0 2024-03-25T18:35:52.761 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.759+0000 7fa90f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:52.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.766+0000 7fa90f7fe640 10 client.?.objecter ms_handle_connect 0x7fa92813f710 2024-03-25T18:35:52.768 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.766+0000 7fa90f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570055177 0 0) 0x7fa928139920 con 0x7fa9280a98f0 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:35:52.771 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.769+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914004eb0 con 0x7fa9280a98f0 2024-03-25T18:35:52.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.772+0000 7fa90f7fe640 10 client.?.objecter ms_handle_connect 0x7fa9281430c0 2024-03-25T18:35:52.773 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.772+0000 7fa90f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:52.779 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268506136 0 0) 0x7fa928138eb0 con 0x7fa9281430c0 2024-03-25T18:35:52.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:35:52.780 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.778+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa928139920 con 0x7fa9281430c0 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.2 v1:172.21.15.176:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994086531 0 0) 0x7fa9280abde0 con 0x7fa92813f710 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 10 monclient(hunting): my global_id is 4763 2024-03-25T18:35:52.784 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.782+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa928138eb0 con 0x7fa92813f710 2024-03-25T18:35:52.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.2 v1:172.21.15.176:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752558356 0 0) 0x7fa928138eb0 con 0x7fa92813f710 2024-03-25T18:35:52.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9280abde0 con 0x7fa92813f710 2024-03-25T18:35:52.803 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.2 v1:172.21.15.176:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa91c0017b0 con 0x7fa92813f710 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): dump: 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536258457 0 0) 0x7fa928139920 con 0x7fa9281430c0 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa928138eb0 con 0x7fa9281430c0 2024-03-25T18:35:52.804 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa920001da0 con 0x7fa9281430c0 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.801+0000 7fa90f7fe640 10 monclient(hunting): dump: 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.805 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.802+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300843230 0 0) 0x7fa914004eb0 con 0x7fa9280a98f0 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.802+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.186:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa928139920 con 0x7fa9280a98f0 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.803+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa9180019b0 con 0x7fa9280a98f0 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.803+0000 7fa90f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.803+0000 7fa90f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.803+0000 7fa90f7fe640 10 monclient(hunting): dump: 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.806 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.807 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2721946814 0 0) 0x7fa928138eb0 con 0x7fa9281430c0 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 >> v1:172.21.15.176:6790/0 conn(0x7fa92813f710 legacy=0x7fa9280ac270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 >> v1:172.21.15.186:6789/0 conn(0x7fa9280a98f0 legacy=0x7fa928138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 1 monclient: found mon.a 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:52.821 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280abb00 con 0x7fa9281430c0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa90f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 10 monclient: _renew_subs 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa928137af0 con 0x7fa9281430c0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 10 monclient: _renew_subs 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281e3690 con 0x7fa9281430c0 2024-03-25T18:35:52.822 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.819+0000 7fa92d263640 1 librados: init done 2024-03-25T18:35:52.823 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.820+0000 7fa92d263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:52.823 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.820+0000 7fa92d263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:52.823 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.820+0000 7fa92d263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:52.823 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.820+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928137ce0 con 0x7fa9281430c0 2024-03-25T18:35:52.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa920003970 con 0x7fa9281430c0 2024-03-25T18:35:52.843 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4008315405 0 0) 0x7fa920003c80 con 0x7fa9281430c0 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 10 monclient: dump: 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:epoch 1 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:fsid ffa5ed0f-6227-4dd7-b2d6-1675fe357662 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:last_changed 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:created 2024-03-25T18:23:25.308958+0000 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:election_strategy: 3 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:0: v1:172.21.15.176:6789/0 mon.a 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:1: v1:172.21.15.186:6789/0 mon.b 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2: v1:172.21.15.176:6790/0 mon.c 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr: 2024-03-25T18:35:52.844 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa9257fa640 4 set_mon_vals no callback set 2024-03-25T18:35:52.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.842+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3104956703 0 0) 0x7fa920019c50 con 0x7fa9281430c0 2024-03-25T18:35:52.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.843+0000 7fa926ffd640 10 monclient: get_auth_request con 0x7fa91406e160 auth_method 0 2024-03-25T18:35:52.845 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.843+0000 7fa90f7fe640 10 client.4728.objecter ms_handle_connect 0x7fa91406e160 2024-03-25T18:35:52.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.845+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1183009562 0 0) 0x7fa920053850 con 0x7fa9281430c0 2024-03-25T18:35:52.847 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.845+0000 7fa90f7fe640 10 client.4728.objecter ms_dispatch 0x7fa928000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:52.848 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.845+0000 7fa90f7fe640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:52.848 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.845+0000 7fa90f7fe640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:52.848 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:52.846+0000 7fa90f7fe640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T18:35:53.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.120+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa92004f360 con 0x7fa9281430c0 2024-03-25T18:35:53.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.120+0000 7fa90f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:53.122 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.120+0000 7fa90f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:53.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.377+0000 7fa92d263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:53.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.377+0000 7fa92d263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:53.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.377+0000 7fa92d263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.176:6789/0 2024-03-25T18:35:53.379 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.377+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa92814f2d0 con 0x7fa9281430c0 2024-03-25T18:35:53.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.411+0000 7fa90f7fe640 1 -- 172.21.15.176:0/1105769292 <== mon.0 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2203406151) 0x7fa920051290 con 0x7fa9281430c0 2024-03-25T18:35:53.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.411+0000 7fa90f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:53.413 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.411+0000 7fa90f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:53.414 INFO:teuthology.misc.health.smithi176.stdout: 2024-03-25T18:35:53.414 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"ffa5ed0f-6227-4dd7-b2d6-1675fe357662","created":"2024-03-25T18:24:25.525914+0000","modified":"2024-03-25T18:34:29.298593+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:29.298593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"946c70d1-508e-40ef-bd1d-a2db16c013ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"dcf2a3a7-6fe4-45dd-8740-3631f2fcbb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"62facedd-b505-4f2b-a30a-86f1e7366d4f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"81d20a92-974e-43cb-b651-9aeb2a692a3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4b761814-fb08-409b-ad5e-8161f0ab123a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d991ea7-a0d8-4dab-95b4-8b314ec586d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f87e7ad2-713e-483d-9d9d-1329acb84493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"700760af-5407-4183-9f2b-1b503a38ccd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:53.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 10 librados: watch_flush enter 2024-03-25T18:35:53.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 10 librados: watch_flush exit 2024-03-25T18:35:53.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T18:35:53.416 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 10 client.4728.objecter successfully canceled tick 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 >> v1:172.21.15.176:6800/2598213983 conn(0x7fa91406e160 legacy=0x7fa914070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 10 monclient: shutdown 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 >> v1:172.21.15.176:6789/0 conn(0x7fa9281430c0 legacy=0x7fa928136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 shutdown_connections 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 >> 172.21.15.176:0/1105769292 conn(0x7fa9280b7ea0 msgr2=0x7fa928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 shutdown_connections 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 -- 172.21.15.176:0/1105769292 wait complete. 2024-03-25T18:35:53.417 INFO:teuthology.misc.health.smithi176.stderr:2024-03-25T18:35:53.415+0000 7fa92d263640 1 librados: shutdown 2024-03-25T18:35:53.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:53.428 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:35:53.429 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:35:53.430 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:35:53.430 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:35:55.362 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:11:24.250 35169== 2024-03-25T18:35:55.362 INFO:tasks.ceph.osd.0.smithi176.stderr:==00:00:11:24.250 35169== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:35:55.402 INFO:tasks.ceph.osd.0.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:35:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:35:59.532 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:35:59.532 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:35:59.533 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:35:59.533 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:01.582 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:11:30.484 35167== 2024-03-25T18:36:01.582 INFO:tasks.ceph.osd.1.smithi176.stderr:==00:00:11:30.484 35167== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:01.623 INFO:tasks.ceph.osd.1.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:04.816 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:06.572 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.864 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:06.620 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.866 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:06.622 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.867 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:06.623 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.877 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:06.634 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.901 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:06.657 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.936 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:06.670 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.939 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:06.673 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:04.956 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:06.691 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:05.635 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:36:05.635 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:36:05.635 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:36:05.635 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:07.598 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:11:36.481 35170== 2024-03-25T18:36:07.598 INFO:tasks.ceph.osd.2.smithi176.stderr:==00:00:11:36.481 35170== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:07.638 INFO:tasks.ceph.osd.2.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:11.738 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:36:11.738 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:36:11.738 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:36:11.739 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:13.664 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:11:42.558 35168== 2024-03-25T18:36:13.664 INFO:tasks.ceph.osd.3.smithi176.stderr:==00:00:11:42.558 35168== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:13.706 INFO:tasks.ceph.osd.3.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:17.840 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:36:17.841 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:36:17.841 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:36:17.841 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:19.567 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:11:48.447 35196== 2024-03-25T18:36:19.567 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:11:48.447 35196== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:19.608 INFO:tasks.ceph.osd.4.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:23.942 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:36:23.943 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:36:23.943 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:36:23.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:25.564 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:11:54.442 35197== 2024-03-25T18:36:25.564 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:11:54.442 35197== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:25.603 INFO:tasks.ceph.osd.5.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:30.045 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:36:30.046 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:36:30.046 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:36:30.046 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:31.633 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:12:00.527 35194== 2024-03-25T18:36:31.633 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:12:00.527 35194== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:31.663 INFO:tasks.ceph.osd.6.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:36.148 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:36:36.148 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:36:36.149 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:36:36.149 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:37.774 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:12:06.668 35195== 2024-03-25T18:36:37.774 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:12:06.668 35195== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:37.812 INFO:tasks.ceph.osd.7.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:39.895 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:41.651 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.908 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:41.664 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.922 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:41.678 34852== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.953 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:41.710 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.955 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:41.711 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.967 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:41.723 34856== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:42.251 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:36:42.251 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:36:42.251 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:36:42.252 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:36:42.252 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:42.253 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:36:42.252+0000 7f9d21160640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 34861) UID: 0 2024-03-25T18:36:42.253 INFO:tasks.ceph.mgr.y.smithi176.stderr:2024-03-25T18:36:42.252+0000 7f9d21160640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:36:42.278 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:36:42.278 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:36:42.278 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:42.302 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:36:42.303 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:36:42.303 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:36:42.303 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:42.332 INFO:tasks.ceph.mon.a.smithi176.stderr:2024-03-25T18:36:42.321+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34821) UID: 0 2024-03-25T18:36:42.334 INFO:tasks.ceph.mon.a.smithi176.stderr:2024-03-25T18:36:42.322+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:36:42.850 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:44.606 34852== 2024-03-25T18:36:42.850 INFO:tasks.ceph.mon.a.smithi176.stderr:==00:00:12:44.606 34852== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:42.859 INFO:tasks.ceph.mon.a.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:48.405 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:36:48.406 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:36:48.406 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:36:48.406 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:48.422 INFO:tasks.ceph.mon.c.smithi176.stderr:2024-03-25T18:36:48.410+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34823) UID: 0 2024-03-25T18:36:48.422 INFO:tasks.ceph.mon.c.smithi176.stderr:2024-03-25T18:36:48.411+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:36:48.959 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:50.715 34856== 2024-03-25T18:36:48.959 INFO:tasks.ceph.mon.c.smithi176.stderr:==00:00:12:50.715 34856== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:48.968 INFO:tasks.ceph.mon.c.smithi176.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:54.508 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:36:54.509 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:36:54.509 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:36:54.509 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:12.133 INFO:tasks.ceph.mon.b.smithi186.stderr:2024-03-25T18:37:12.125+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34644) UID: 0 2024-03-25T18:37:12.133 INFO:tasks.ceph.mon.b.smithi186.stderr:2024-03-25T18:37:12.127+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:37:12.784 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:14.518 34648== 2024-03-25T18:37:12.784 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:14.518 34648== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:12.794 INFO:tasks.ceph.mon.b.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:12.813 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:37:12.814 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:37:12.814 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:37:12.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:37:12.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:37:12.863 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:37:12.923 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:37:12.924 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:37:12.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:37:12.988 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:37:13.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:37:13.055 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:37:13.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:37:13.115 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:37:13.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:37:13.223 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:37:13.311 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:37:13.312 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:37:13.371 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:37:13.371 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:37:13.438 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:37:13.440 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/data/mon.a.tgz 2024-03-25T18:37:13.440 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-25T18:37:13.456 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.H0qJ7lo9gO 2024-03-25T18:37:13.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.H0qJ7lo9gO 2024-03-25T18:37:13.561 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.H0qJ7lo9gO 2024-03-25T18:37:13.647 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.H0qJ7lo9gO is 134KB 2024-03-25T18:37:13.699 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.H0qJ7lo9gO 2024-03-25T18:37:13.714 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/data/mon.c.tgz 2024-03-25T18:37:13.715 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-25T18:37:13.767 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.c0egxqqgwy 2024-03-25T18:37:13.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.c0egxqqgwy 2024-03-25T18:37:13.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.c0egxqqgwy 2024-03-25T18:37:13.937 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.c0egxqqgwy is 134KB 2024-03-25T18:37:13.989 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.c0egxqqgwy 2024-03-25T18:37:14.004 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/data/mon.b.tgz 2024-03-25T18:37:14.004 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-25T18:37:14.020 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.Yjd4wxD76w 2024-03-25T18:37:14.020 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Yjd4wxD76w 2024-03-25T18:37:14.126 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.Yjd4wxD76w 2024-03-25T18:37:14.213 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.Yjd4wxD76w is 131KB 2024-03-25T18:37:14.264 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.Yjd4wxD76w 2024-03-25T18:37:14.278 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:37:14.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:14.281 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:14.665 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:14.906 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:37:14.908 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi176/crash 2024-03-25T18:37:14.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:14.936 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi186/crash 2024-03-25T18:37:14.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:14.964 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:37:14.965 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:14.981 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35166.log 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41990.log 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.35166.log: /var/log/ceph/ceph-client.admin.41990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41717.log 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35166.log.gz 2024-03-25T18:37:15.049 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39101.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41990.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40819.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39882.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41717.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39023.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39101.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39882.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40819.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40311.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39882.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40546.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39023.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2024-03-25T18:37:15.050 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40546.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40311.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36331.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41522.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40546.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.36331.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/mon.c.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36331.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:37:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/mon.a.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41288.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39531.log 86.2% 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40233.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40780.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39999.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39765.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39531.log.gz 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-03-25T18:37:15.052 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39999.log: /var/log/ceph/ceph-client.admin.39765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40233.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39843.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39999.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39765.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.37393.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41249.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39453.log 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2024-03-25T18:37:15.053 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40077.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38945.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40077.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39453.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40077.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40585.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38945.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40624.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40585.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40624.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41600.log 2024-03-25T18:37:15.054 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29012.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40624.log.gz 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.tmp-client.admin.29012.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29012.log.gz 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34863.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41600.log.gz 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39140.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34863.log.gzgzip 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-03-25T18:37:15.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39140.log: 87.2%/var/log/ceph/ceph-client.admin.40428.log: -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40663.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39140.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40663.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.1.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40663.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41795.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41444.log: -- /var/log/ceph/ceph-client.admin.39921.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39921.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41795.log.gz 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2024-03-25T18:37:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41561.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41366.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38984.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41561.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40155.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41366.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38984.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38906.log 2024-03-25T18:37:15.057 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40155.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41873.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39570.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40936.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38906.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39570.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38826.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41873.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41327.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39570.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40936.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41327.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38826.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.38867.log: gzip -5 --verbose 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41327.log.gz 2024-03-25T18:37:15.058 INFO:teuthology.orchestra.run.smithi176.stderr: -- /var/log/ceph/ceph-client.admin.39336.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40194.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39804.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40194.log: gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: /var/log/ceph/ceph-client.admin.39297.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39804.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40194.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39297.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39804.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39258.log: -- /var/log/ceph/ceph-client.admin.41756.log 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:37:15.059 INFO:teuthology.orchestra.run.smithi176.stderr: 86.2%/var/log/ceph/ceph-client.admin.41756.log: -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41210.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.39414.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39414.log: /var/log/ceph/ceph-client.admin.41054.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39414.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41951.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41132.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41951.log: gzip 86.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz -- 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: /var/log/ceph/ceph-client.admin.40350.log 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41951.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:37:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40350.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40038.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.35433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40468.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39726.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.40468.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40468.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39492.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39726.log: /var/log/ceph/ceph-client.admin.35268.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39726.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41014.log 2024-03-25T18:37:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39492.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38578.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39179.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41014.log: /var/log/ceph/ceph-client.admin.38578.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41014.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38578.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39062.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39179.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39179.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39062.log: /var/log/ceph/ceph-client.admin.40897.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39062.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39375.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.41834.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:37:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.39375.log: /var/log/ceph/ceph-client.admin.41639.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39375.log.gz 2024-03-25T18:37:15.063 INFO:teuthology.orchestra.run.smithi176.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2024-03-25T18:37:15.065 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:37:15.065 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35004.log 2024-03-25T18:37:15.065 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34844.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.34844.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35004.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.34804.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34844.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.5.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/mon.b.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34884.log 2024-03-25T18:37:15.066 INFO:teuthology.orchestra.run.smithi186.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35095.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.34884.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34924.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34884.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35095.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.34924.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34924.log.gz 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34964.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:37:15.067 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.34964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35046.log 2024-03-25T18:37:15.068 INFO:teuthology.orchestra.run.smithi186.stderr: 87.2%/var/log/ceph/ceph-mon.b.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34964.log.gz 2024-03-25T18:37:15.068 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:37:15.068 INFO:teuthology.orchestra.run.smithi186.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:37:15.068 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35046.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35046.log.gz 2024-03-25T18:37:15.104 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.c.log: 92.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:37:15.859 INFO:teuthology.orchestra.run.smithi176.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:37:15.866 INFO:teuthology.orchestra.run.smithi176.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:37:15.897 INFO:teuthology.orchestra.run.smithi186.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:37:15.906 INFO:teuthology.orchestra.run.smithi186.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:37:15.916 INFO:teuthology.orchestra.run.smithi186.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:37:16.168 INFO:teuthology.orchestra.run.smithi176.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:37:16.170 INFO:teuthology.orchestra.run.smithi176.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:37:16.218 INFO:teuthology.orchestra.run.smithi186.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:37:20.133 INFO:teuthology.orchestra.run.smithi176.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:37:20.277 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:37:20.279 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-25T18:37:20.279 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m5.257s 2024-03-25T18:37:20.279 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m8.742s 2024-03-25T18:37:20.279 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.318s 2024-03-25T18:37:20.314 INFO:teuthology.orchestra.run.smithi176.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:37:20.316 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-25T18:37:20.316 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m5.324s 2024-03-25T18:37:20.317 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m13.965s 2024-03-25T18:37:20.317 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.575s 2024-03-25T18:37:20.317 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:37:20.317 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi176/log 2024-03-25T18:37:20.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:22.774 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi186/log 2024-03-25T18:37:22.775 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:24.259 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:24.461 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d46620b3e67f4a838b27cf3f63ce93dd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:24.465 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:37:24.475 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:37:24.489 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:37:24.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:24.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:24.579 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:37:24.587 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:37:24.587 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:24.590 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:24.604 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:24.607 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:24.608 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:24.608 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-25T18:37:24.608 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +527us[ +527us] +/- 75ms 2024-03-25T18:37:24.608 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -1103us[-1097us] +/- 60ms 2024-03-25T18:37:24.608 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +757us[ +757us] +/- 65ms 2024-03-25T18:37:24.609 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:24.612 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:24.612 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-25T18:37:24.612 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +535us[ +528us] +/- 75ms 2024-03-25T18:37:24.612 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -1089us[-1095us] +/- 60ms 2024-03-25T18:37:24.612 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +777us[ +770us] +/- 65ms 2024-03-25T18:37:24.613 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:24.613 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:37:24.621 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:37:24.622 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:37:24.630 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:24.719 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:24.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:24.821 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:24.852 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:37:24.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:37:24.870 INFO:teuthology.task.internal:Duration was 8559.238092 seconds 2024-03-25T18:37:24.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:37:24.878 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:37:24.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:24.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:24.922 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:24.937 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:25.378 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:37:25.378 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-03-25T18:37:25.378 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:25.399 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-03-25T18:37:25.399 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:25.424 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:37:25.425 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:25.442 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:25.494 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:37:25.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:25.511 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:25.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:37:25.652 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:37:25.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:25.679 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:25.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:37:25.715 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:25.722 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:25.745 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-03-25T18:37:25.770 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-03-25T18:37:25.791 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:25.840 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:25.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:37:25.864 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:37:25.865 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi176 2024-03-25T18:37:25.865 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:25.917 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252/remote/smithi186 2024-03-25T18:37:25.917 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:25.952 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:37:25.953 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:25.955 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:26.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:37:26.012 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:37:26.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:37:26.021 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:37:26.021 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:26.023 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:26.035 INFO:teuthology.orchestra.run.smithi176.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:26.056 INFO:teuthology.orchestra.run.smithi186.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:26.057 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:37:26.066 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:37:26.085 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:37:26.112 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-25T18:37:26.113 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:26.137 DEBUG:teuthology.nuke:shortname: smithi186 2024-03-25T18:37:26.138 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:26.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:04:44.478006', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK6Y5chr1P3W+bPK6Xaeaip+/nsTZQ5qzyhjymkTeb+IGibowaAwcZ9ftmZrfeqMR0LnOXCZWuKsj9ExgiCMKQ='} 2024-03-25T18:37:26.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:04:44.479216', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfSSRsp6KvzxWvC/Wea72U/E4M7bNUtEnuPpOpE7RjfVCjbi2dR1vm7niezXKK3+lqWOkBM70RbsK8dCV/pf4I='} 2024-03-25T18:37:26.191 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-25T18:37:26.192 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:26.215 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:26.216 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:26.230 INFO:teuthology.orchestra.console.smithi186:Power off 2024-03-25T18:37:26.230 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:26.243 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-25T18:37:26.252 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:26.252 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:26.274 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-25T18:37:30.244 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:30.275 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:30.355 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-25T18:37:30.384 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-25T18:37:34.361 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:34.386 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:34.473 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-25T18:37:34.495 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-03-25T18:37:34.574 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-25T18:37:34.596 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-03-25T18:37:34.700 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 8559.238091945648 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d46620b3e67f4a838b27cf3f63ce93dd status: fail success: false 2024-03-25T18:37:34.701 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:37:34.792 INFO:teuthology.run:FAIL